This notebook is neither clean nor consistent, stemming not only from my limited experience with Jupyter notebooks but also from a strong underlying reason. There are two conflicting tasks.
I am writing and designing software for 13 years. Code is there to solve some user problem, not to show off cool coding techniques... I am striving for this here, therefore in some cases I'm just igonring second goal... but still trying to show, that, yes I can do complex code if required.
Before submitting this to any business or manager I'd remove most of the code from the middle part, just leaving problem statement, crucial pieces of data analysis, best model, and summary. Current version is too long and too full of irrelevant technical details to have good presentational value.
Peer review and validation of math and domain specific assumptions is mandatory.
The context: Why is this problem important to solve?
State of affairs: How is malaria detected presently?
The objectives: What is the intended goal?
The key questions: What are the key questions that need to be answered?
The problem formulation: What is it that we are trying to solve using data science?
TECHNICAL NOTE
Since this is a school project, it includes code that may not be necessary for a real-world study. This is to demonstrate my ability to tackle tasks, which are irrelevant to the central question of this project.
There are a total of 24,958 train and 2,600 test images (colored) that we have taken from microscopic images. These images are of the following categories:
Parasitized: The parasitized cells contain the Plasmodium parasite which causes malaria
Uninfected: The uninfected cells are free of the Plasmodium parasites
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import zipfile
import os
import glob
from collections import Counter
import cv2
from tqdm import tqdm
import random
import h5py
from sklearn.utils import shuffle
import matplotlib.colors as mcolors
import tensorflow as tf
from tensorflow.keras.models import Sequential, Model, load_model
from tensorflow.keras.backend import clear_session
from tensorflow.keras.layers import Conv2D, ReLU, LeakyReLU, PReLU, MaxPooling2D, Flatten, Dense, BatchNormalization, Dropout
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
from sklearn.metrics import classification_report, confusion_matrix
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.utils import register_keras_serializable
from tensorflow.keras.applications import VGG16
import json
Here are various functions and variables used multiple times later in code. Defining them upfront they can be compiled in one place, and their definitions doesn't intterrupt logic of project and finally this call can be run to allow running multiple sections independently (as running everything step by stap is painfully slow, especially considering not so uncommon interruptions in backend service provided by Google)
project_path = "/content/drive/MyDrive/projects/school/Capstone Project/"
test_data_folder = project_path + "data/cell_images/test/"
train_data_folder = project_path + "data/cell_images/train/"
test_data_path_high_res = project_path + "test_data.hdf5"
train_data_path_high_res = project_path + "training_data.hdf5"
test_data_file_8x8 = project_path + "test_data_8x8.npz"
train_data_file_8x8 = project_path + "train_data_8x8.npz"
infected = "parasitized"
healthy = "uninfected"
# No more used, I played with it, but results were very poor
def read_data_8x8(npz_file_path):
""" Load images and labels of 8x8 format """
with np.load(npz_file_path) as data:
images = data['images']
labels = data['labels']
return images, labels
# Helper to get labels from given data set
def read_labels_high_res(file_path):
with h5py.File(file_path, 'r') as hf:
labels = np.array(hf['labels'])
return labels
# Helper to get images from given data set
def read_images_high_res(file_path):
with h5py.File(file_path, 'r') as hf:
images = np.array(hf['images'])
return images
def set_random_seeds():
"""Sets the seeds for numpy and tensorflow"""
destiny = 44
random.seed(destiny)
np.random.seed(destiny)
tf.random.set_seed(destiny)
def model_file_path(model_name):
return os.path.join(project_path, f"{model_name}.keras")
def model_history_file_path(model_name):
return os.path.join(project_path, f"{model_name}_history.json")
def save_history(history, model_name):
path = model_history_file_path(model_name)
with open(path, 'w') as history_file:
json.dump(history.history, history_file)
print(f"Training history saved to {path}")
# Reads history from file
def read_history(model_name):
path = model_history_file_path(model_name)
with open(path, 'r') as f:
history = json.load(f)
return history
def plot_accuracy(model_name):
"""Plots training and validation accuracy from model history"""
model_history = read_history(model_name)
acc = model_history['accuracy']
val_acc = model_history['val_accuracy']
epochs = range(1, len(acc) + 1)
plt.figure(figsize=(10, 6))
plt.plot(epochs, acc, 'bo-', label='Training accuracy')
plt.plot(epochs, val_acc, 'ro-', label='Validation accuracy')
plt.title('Training and Validation Accuracies')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()
plt.show()
def show_confusion(true, predicted):
"""Plots confusion matrix for classification results"""
cm = confusion_matrix(true, predicted)
classes = ['Healthy', 'Infected']
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=classes, yticklabels=classes)
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
# I print only accuracy as on research stage it doesn't matter what is sensitivity and specificyt
# When building final model we'd need to carefully tweak them with respect to domain requirements
# But it's relatively easy by adjusting threshold value
def predictions(model_name, test_images, test_labels):
"""Simple function to run predict on the saved model and binarize output"""
predictor = load_model(model_file_path(model_name))
_, test_acc, test_auc = predictor.evaluate(test_images, test_labels, verbose=0)
print(f'Test accuracy: {test_acc}')
predictions = predictor.predict(test_images)
return np.where(predictions > 0.5, 1, 0)
def compile_and_fit_model(model, model_name, images, labels, validation_split=0.25, epochs=25, batch_size=512, learning_rate=0.001):
"""Helper function to run all the models to avoid copy-pasting code"""
set_random_seeds() # to ensure reproductability
model.compile(
loss='binary_crossentropy',
optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate),
metrics=['accuracy', tf.keras.metrics.AUC()]
)
model_path = model_file_path(model_name)
history = model.fit(
images, labels,
validation_split = validation_split,
epochs=epochs,
batch_size=batch_size,
callbacks=[
EarlyStopping(monitor='val_loss', patience=10, verbose=1, mode='min'),
ModelCheckpoint(model_path, monitor='val_loss', save_best_only=True, mode='min', verbose=1)])
save_history(history, model_name)
return history
Note:
data_folder = project_path + "data"
# The assumption is that there won't be new data, so once extracted to persistent storage
# I won't perform this again - therefore I'm not checking if content is full and up to date
if not os.path.exists(data_folder):
os.makedirs(data_folder)
with zipfile.ZipFile(project_path + "cell_images.zip", 'r') as zip_ref:
zip_ref.extractall(data_folder)
# Checking if data was unzipped sucessfully and we have all images as expected
def get_files_recursively(folder_path):
all_items = glob.glob(os.path.join(folder_path, '**', '*'), recursive=True)
return [item for item in all_items if os.path.isfile(item)]
def check_n_of_files(expected_n_of_files, folder):
actual_n_of_files = len(get_files_recursively(folder))
print(f"[{actual_n_of_files == expected_n_of_files}] - there are {actual_n_of_files} in {folder}")
check_n_of_files(24958, train_data_folder)
check_n_of_files(2600, test_data_folder)
[True] - there are 24958 in /content/drive/MyDrive/projects/school/Capstone Project/data/cell_images/train/ [True] - there are 2600 in /content/drive/MyDrive/projects/school/Capstone Project/data/cell_images/test/
The extracted folder has different folders for train and test data will contain the different sizes of images for parasitized and uninfected cells within the respective folder name.
The size of all images must be the same and should be converted to 4D arrays so that they can be used as an input for the convolutional neural network. Also, we need to create the labels for both types of images to be able to train and test the model.
Let's do the same for the training data first and then we will use the same code for the test data as well.
# First let's see what are the resolutions of images and if they all consist the same number of layers
def get_image_properties(files):
properties_list = []
for file_path in files:
image = tf.io.read_file(file_path)
image = tf.image.decode_image(image, channels=3, expand_animations=False)
height, width, channels = image.shape
extension = os.path.splitext(file_path)[1] # File extension
properties = (width, height, channels, extension)
properties_list.append(properties)
return properties_list
image_properties = get_image_properties(get_files_recursively(data_folder))
# Check if each is .png and has 3 layers
files_with_different_n_of_layers = 0
files_in_different_format = 0
for img_props in image_properties:
if img_props[2] != 3:
files_with_different_n_of_layers += 1
if img_props[3] != '.png':
files_in_different_format += 1
print(f"There are {files_with_different_n_of_layers} files with different number of layers")
print(f"There are {files_in_different_format} files in non .png format")
There are 0 files with different number of layers There are 0 files in non .png format
# Now it's confirmed that they are all .png images with 3 layers, let's check their sizes
images_below_64 = 0
images_below_128 = 0
images_below_256 = 0
images_above_256 = 0
for width, height, _, _ in image_properties:
if min(width, height) < 64:
images_below_64 += 1
elif min(width, height) < 128:
images_below_128 += 1
elif min(width, height) < 256:
images_below_256 += 1
else:
images_above_256 += 1
print(f"{images_below_64} images are below 64x64 pixels")
print(f"{images_below_128} images are between 64x64 and 128x128 pixels")
print(f"{images_below_256} images are between 128x128 and 256x256 pixels")
print(f"{images_above_256} images are above 256x256 pixels")
20 images are below 64x64 pixels 16178 images are between 64x64 and 128x128 pixels 11357 images are between 128x128 and 256x256 pixels 3 images are above 256x256 pixels
There are no labels given - I'll generate them when preparing data, so this point is invalid for this data. Yes - I need to check how many there are images in each set to see if it's balanced but it's mentioned later and I'm not yet done with preprocessing data.
# inapporapriate for this data
Images have very different sizes and aspects ratio. Taking into consideration their distribution I'll convert them to uniform size of 64x64 pixels what is still reasonable. Quite maybe lower resolution like 32x32 would be good enough but given the goal of this task we are to check if accurate classification is possible, not focus on opitimizing performance and computational cost, so starting with higer resolution is most reasonable approach.
def check_pixel_value_ranges(folder, data_title):
file_paths = get_files_recursively(folder)
min_val, max_val = 255, 0 # Initialize to opposite extremes
for file_path in file_paths:
img = tf.io.read_file(file_path)
img = tf.image.decode_image(img, expand_animations=False)
min_val = min(min_val, tf.reduce_min(img))
max_val = max(max_val, tf.reduce_max(img))
print(f"{data_title} pixel values are in range: {min_val.numpy()} - {max_val.numpy()}")
check_pixel_value_ranges(train_data_folder, "Training data")
check_pixel_value_ranges(test_data_folder, "Test data")
Training data pixel values are in range: 0 - 255 Test data pixel values are in range: 0 - 255
It's confirmed that .png files are according to specification and pixel values are encoded 8bits per channel.
def print_data_balance(data_folder, data_title):
infected_images = len(get_files_recursively(data_folder + "/" + infected))
healthy_images = len(get_files_recursively(data_folder + "/" + healthy))
ratio = round((infected_images / healthy_images))
print(f"In {data_title} there are {infected_images} images of infected cells and {healthy_images} images of healthy cells. Ratio is {ratio}")
print_data_balance(test_data_folder, "Test Data")
print_data_balance(train_data_folder, "Training Data")
In Test Data there are 1300 images of infected cells and 1300 images of healthy cells. Ratio is 1 In Training Data there are 12582 images of infected cells and 12376 images of healthy cells. Ratio is 1
# One function that takes file_path as input and returns a it in format that may be fed into CNN
# It will be used to process data for training but also would be part of prediction program
def preprocess_image(file_path):
"""This function will convert image to uniform way:
- 64x64 pixels using aspect fill to don't disturb data unitentionally
- 0-1 pixel values
- HSV color space"""
img = cv2.imread(file_path, cv2.IMREAD_COLOR)
# Resize the image to maintain aspect ratio
height, width = img.shape[:2]
ratio = min(64 / height, 64 / width)
new_shape = (int(width * ratio), int(height * ratio))
img = cv2.resize(img, new_shape, interpolation=cv2.INTER_AREA)
# Pad the image to 64x64
delta_height = 64 - new_shape[1]
delta_width = 64 - new_shape[0]
top, bottom = delta_height // 2, delta_height - (delta_height // 2)
left, right = delta_width // 2, delta_width - (delta_width // 2)
img = cv2.copyMakeBorder(img, top, bottom, left, right, cv2.BORDER_CONSTANT, value=[0, 0, 0])
# Convert the image to HSV color space and normalize pixel values
img = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
img = img.astype('float32') / 255.0
return img
# Data shouldn't be ordered so batches aren't full of one type of label
# So let's create labels and paths and then shuffle them so we can process the files
def prepare_labeled_and_shuffled_paths(data_dir):
categories=(healthy, infected)
paths_with_labels = []
for category in categories:
class_num = categories.index(category)
path = os.path.join(data_dir, category)
files = os.listdir(path)
for img_name in files:
file_path = os.path.join(path, img_name)
paths_with_labels.append((file_path, class_num))
# Shuffle the list to mix up the order of files and their corresponding labels
random.shuffle(paths_with_labels)
return paths_with_labels
# Preparing data is slow, so it's nice to have it stored in format ready to use,
# when getting back to the project after session expired for one reason or another
def perepare_data_file(input_folder, output_file):
"""
Preprocess images, shuffle them, and save along with labels to a .npz file.
Parameters:
- data_dir: Path to the data directory, which contains subdirectories for each class.
- output_file_path: Path where the file will be stored
"""
# Preapre input data that will be not ordered by labels
files_and_labels = prepare_labeled_and_shuffled_paths(input_folder)
# Paths are standardized to the same length so it's easier to store and navigate data
max_path_length = max(len(path) for path, _ in files_and_labels)
with h5py.File(output_file, 'w') as hf:
# Create datasets with fixed sizes
images = hf.create_dataset('images', shape=(len(files_and_labels), 64, 64, 3), dtype=np.float32)
labels = hf.create_dataset('labels', shape=(len(files_and_labels),), dtype=np.int8)
paths = hf.create_dataset('paths', (len(files_and_labels),), dtype=f'S{max_path_length}')
for i, (file_path, label) in enumerate(tqdm(files_and_labels, desc="Processing images")):
processed_img = preprocess_image(file_path)
images[i] = processed_img # Directly store the 64x64x3 image
labels[i] = label
standardized_path = file_path.ljust(max_path_length) # Standardize path length by padding with spaces
paths[i] = np.string_(standardized_path.encode('utf-8')) # Ensure proper encoding
perepare_data_file(test_data_folder, test_data_path_high_res)
Processing images: 100%|██████████| 2600/2600 [00:39<00:00, 66.50it/s]
perepare_data_file(train_data_folder, train_data_path_high_res)
Processing images: 100%|██████████| 24958/24958 [10:56<00:00, 38.01it/s]
# Not sure what to code here - all is ready
def plot_class_distribution(train_labels, test_labels):
# Count the occurrence of each class in the datasets
unique, train_counts = np.unique(train_labels, return_counts=True)
_, test_counts = np.unique(test_labels, return_counts=True)
# Plotting
classes = ['Healthy', 'Infected']
fig, ax = plt.subplots(figsize=(7, 3.5))
index = np.arange(len(classes))
bar_width = 0.2
train_bars = ax.bar(index - bar_width/2, train_counts, bar_width, label='Train')
test_bars = ax.bar(index + bar_width/2, test_counts, bar_width, label='Test')
ax.set_ylabel('Samples Count')
ax.set_title('Class Distribution in Train and Test Sets')
ax.set_xticks(index)
ax.set_xticklabels(classes)
ax.legend()
plt.tight_layout()
plt.show()
train_labels = read_labels_high_res(train_data_path_high_res)
test_labels = read_labels_high_res(test_data_path_high_res)
plot_class_distribution(train_labels, test_labels)
Just a visual confirmation for people who may prefer images than the numbers. As we previously checked data is almost perfectly balanced.
def display_images_6_x_6(healthy_images, infected_images):
fig, axs = plt.subplots(6, 6, figsize=(12, 12))
axs = axs.ravel() # Flatten the array of axes for easy iteration
all_images = healthy_images + infected_images
for idx, img in enumerate(all_images):
axs[idx].imshow(img)
category_label = 'Healthy' if idx < len(healthy_images) else 'Infected'
axs[idx].set_title(category_label)
axs[idx].axis('off')
plt.tight_layout()
plt.show()
def load_original_images(how_many, what_category):
base_path = "/content/drive/MyDrive/projects/school/Capstone Project/data/cell_images/train/"
folder_path = os.path.join(base_path, what_category)
image_paths = []
image_names = os.listdir(folder_path)[:how_many]
image_paths.extend([os.path.join(folder_path, name) for name in image_names])
images = []
for img_path in image_paths:
img = cv2.imread(img_path)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # Convert BGR to RGB
images.append(img)
return images
org_18_healthy_images = load_original_images(18, healthy)
org_18_infected_images = load_original_images(18, infected)
display_images_6_x_6(org_18_healthy_images, org_18_infected_images)
It looks too simple. Basically detecting if there is little violet in the image and not all is pink would be enough to distingiquish infected cells. That's something I'd double check with specialist if I had access to any.
def load_preprocessed_images(how_many, what_category):
labels = read_labels_high_res(test_data_path_high_res)
label_value = what_category != healthy
indices = np.where(labels == label_value)[0]
indices = indices[:how_many]
images = []
with h5py.File(test_data_path_high_res, 'r') as data_file:
for idx in indices:
img = np.array(data_file['images'][idx])
images.append(img)
return images
p_h_images = load_preprocessed_images(18, healthy)
p_i_images = load_preprocessed_images(18, infected)
display_images_6_x_6(p_h_images, p_i_images)
Problem: Data and labels are inconsistent. Labels are binary, infected/not infected, but images are precise with relatively high resolution (almost all of them are above 64x64 pixels) what is typical for good quality aparature and thin smear microscope analysis. But this method (thin smear) is used to learn about details of infection. For binary classification if malaria is present usually much different data is used - thick blood smears with much lower zoom level (10-40). In such case single cell is very small. Especially if it's to be widely used and applied to labs where equipment isn't top tier. In those cases single cell is just few by few pixels. Additionally cells are not well separated from each other. But thick smears allows to see much more cells and such tests have much more sensitive. Additionally thick smears are easier to prepare so this method is more applicable for widespread use. Additinally looking at the images shows that aspect ratio shouldn't be an issue, so we need to rescale the data to keep most information (aspect fit instead of aspect fill).
Correction of assumptions If this project has to resemble any real research problem we must try to detect presence of infection on as small images as possible - 8x8 is max we can assume. Additionally it must be really fast. In real world there may be well over thousand cells on single microscope image, so we would use analysis of single a lot for every single sample. It must be efficient.
Note At this stage we are hand-waving problem that in thick smear cells overlap, there are many layers of them. Probably best way would be to train model on whole sample images, not on single cells, but we have to work with what we have.
# There are 1300 images for each category in Test data
# That should be enough, so we can save some time and do that on test data instead of
# computating much bigger training data set (around 13k images per category)
labels = read_labels_high_res(test_data_path_high_res)
def create_mean_image_from_test_set(label_value):
indices = np.where(labels == label_value)[0]
with h5py.File(test_data_path_high_res, 'r') as data_file:
images = data_file['images']
return np.mean(np.array([images[i] for i in indices]), axis=0)
def display_image(img, title):
img_rgb = mcolors.hsv_to_rgb(img)
fig, axs = plt.subplots(1, 2, figsize=(10, 5))
axs[0].imshow(img)
axs[0].set_title(title + ' (HSV)')
axs[0].axis('off')
axs[1].imshow(img_rgb)
axs[1].set_title(title + ' (RGB)')
axs[1].axis('off')
plt.show()
Mean image for parasitized
mean_of_infected = create_mean_image_from_test_set(1)
display_image(mean_of_infected, "Infected")
Mean image for uninfected
mean_of_healthy = create_mean_image_from_test_set(0)
display_image(mean_of_healthy, "Healty")
Since we have both means, let's mean difference
mean_difference = abs(mean_of_healthy - mean_of_infected)
display_image(mean_difference, "Infected - Healthy")
def display_separate_channels(img, colors='hsv'):
hue, saturation, value = mean_difference[:,:,0], mean_difference[:,:,1], mean_difference[:,:,2]
fig, axs = plt.subplots(1, 3, figsize=(15, 5))
def display_one_layer(channel, subplot, title):
if subplot == 0:
colorspace = colors
else:
colorspace = colors
axs[subplot].imshow(channel, cmap=colorspace)
axs[subplot].set_title(title)
axs[subplot].axis('off')
display_one_layer(img[:,:,0], 0, "Hue")
display_one_layer(img[:,:,1], 1, "Saturation")
display_one_layer(img[:,:,2], 2, "Value")
plt.show()
display_separate_channels(mean_of_healthy)
display_separate_channels(mean_of_infected)
display_separate_channels(mean_difference, colors='gray')
Assumption: visual inspection should be done by someone wiht deep biological knowledge. That said there are some clear visual differences. Interesting insights:
# Already done
# Already done
HSV format seems to be better fit for this task. Hue variations seems to carry most information. So maybe, quite maybe we could work with only this one layer. If single image conatins over 1000 cells we need something fast, so any data that can be discarded needs to be discarded in preprocessing.
def blur_images(images_to_be_blurred):
blurred = []
for img in images_to_be_blurred:
blurry_img = cv2.GaussianBlur(img, (5, 5), sigmaX=1)
blurred.append(blurry_img)
return blurred
some_images = load_preprocessed_images(18, infected)
other_images = blur_images(some_images)
display_images_6_x_6(some_images, other_images)
# I'm not convinced I should follow this path
Think About It: Would blurring help us for this problem statement in any way? What else can we try?
Outdated: I tried 8x8 resolution, but accuracy was around 60% so it's not of any use.
Last but not least, let's preapre some low resolution data to figure out how model works with it.
# This is just to make data loading fast
def preprocess_image_8x8(file_path):
"""This function will convert image to uniform way:
- 8x8 pixels without preserving aspect ratio to utilize all pixels
- No other changes, transforming color, standardizing values will be part of model"""
img = cv2.imread(file_path, cv2.IMREAD_COLOR)
img = cv2.resize(img, (8, 8), interpolation=cv2.INTER_AREA)
return img
def prepare_data_8x8(input_folder, output_file):
"""
Preprocess images, shuffle them, and save along with labels to a .npz file.
Parameters:
- data_dir: Path to the data directory, which contains subdirectories for each class.
- output_file_path: Path where the file will be stored
"""
image_data = []
labels = []
print("preparing: ", input_folder)
# Loop through each subdirectory in the input folder
for subdir in sorted(os.listdir(input_folder)):
subdir_path = os.path.join(input_folder, subdir)
if os.path.isdir(subdir_path):
# Determine the label based on the subdirectory name
label = int(subdir == infected)
for fname in os.listdir(subdir_path):
file_path = os.path.join(subdir_path, fname)
if os.path.isfile(file_path):
img = preprocess_image_8x8(file_path)
image_data.append(img)
labels.append(label)
# Convert lists to numpy arrays and shouffle it
image_data = np.array(image_data, dtype=np.float32)
labels = np.array(labels, dtype=np.int8)
image_data, labels = shuffle(image_data, labels, random_state=44)
# Save the processed dataset to a .npz file for easy loading
np.savez(output_file, images=image_data, labels=labels)
print(f'Dataset saved to {output_file}.npz with {len(image_data)} images and binary labels.')
prepare_data_8x8(test_data_folder, test_data_file_8x8)
preparing: /content/drive/MyDrive/projects/school/Capstone Project/data/cell_images/test/ Dataset saved to /content/drive/MyDrive/projects/school/Capstone Project/test_data_8x8.npz.npz with 2600 images and binary labels.
prepare_data_8x8(train_data_folder, train_data_file_8x8)
preparing: /content/drive/MyDrive/projects/school/Capstone Project/data/cell_images/train/ Dataset saved to /content/drive/MyDrive/projects/school/Capstone Project/train_data_8x8.npz.npz with 24958 images and binary labels.
train_images_64x64 = read_images_high_res(train_data_path_high_res)
train_labels_64x64 = read_labels_high_res(train_data_path_high_res)
test_images_64x64 = read_images_high_res(test_data_path_high_res)
test_labels_64x64 = read_labels_high_res(test_data_path_high_res)
# No more used - I played with it but accuracy of around 60% was dramatically too low
test_images_8x8, test_labels_8x8 = read_data_8x8(test_data_file_8x8)
train_images_8x8, train_labels_8x8 = read_data_8x8(train_data_file_8x8)
Note: The Base Model has been fully built and evaluated with all outputs shown to give an idea about the process of the creation and evaluation of the performance of a CNN architecture. A similar process can be followed in iterating to build better-performing CNN architectures.
# Loaded all them in one place at the beginning
# Already done - DRY principle
I'll make very simple CNN model, just to get grasp how it works, what are the results, get the feel of how this data works with NN.
model_1 = Sequential([
Conv2D(6, kernel_size=(3, 3), padding='valid', input_shape=(64, 64, 3)),
PReLU(),
MaxPooling2D(pool_size=(2, 2)),
Conv2D(4, kernel_size=(3, 3), padding='valid'),
PReLU(),
MaxPooling2D(pool_size=(2, 2)),
Flatten(),
Dense(16),
PReLU(),
Dense(1, activation='sigmoid')
])
# Function defined at the top to not repeat the same code again and agian (DRY principle)
Using Callbacks
# Function defined at the top to not repeat the same code again and agian (DRY principle)
Fit and train our Model
compile_and_fit_model(
model=model_1,
model_name="model_1",
images=train_images_64x64,
labels=train_labels_64x64,
epochs=20,
batch_size=512)
Epoch 1/20 37/37 [==============================] - ETA: 0s - loss: 0.6374 - accuracy: 0.6507 - auc_8: 0.6989 Epoch 1: val_loss improved from inf to 0.60156, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 60s 2s/step - loss: 0.6374 - accuracy: 0.6507 - auc_8: 0.6989 - val_loss: 0.6016 - val_accuracy: 0.6994 - val_auc_8: 0.7454 Epoch 2/20 37/37 [==============================] - ETA: 0s - loss: 0.5965 - accuracy: 0.6932 - auc_8: 0.7499 Epoch 2: val_loss improved from 0.60156 to 0.58442, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 51s 1s/step - loss: 0.5965 - accuracy: 0.6932 - auc_8: 0.7499 - val_loss: 0.5844 - val_accuracy: 0.6994 - val_auc_8: 0.7666 Epoch 3/20 37/37 [==============================] - ETA: 0s - loss: 0.5773 - accuracy: 0.7043 - auc_8: 0.7762 Epoch 3: val_loss improved from 0.58442 to 0.56946, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 42s 1s/step - loss: 0.5773 - accuracy: 0.7043 - auc_8: 0.7762 - val_loss: 0.5695 - val_accuracy: 0.7054 - val_auc_8: 0.7889 Epoch 4/20 37/37 [==============================] - ETA: 0s - loss: 0.5636 - accuracy: 0.7165 - auc_8: 0.7912 Epoch 4: val_loss improved from 0.56946 to 0.55460, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 41s 1s/step - loss: 0.5636 - accuracy: 0.7165 - auc_8: 0.7912 - val_loss: 0.5546 - val_accuracy: 0.7226 - val_auc_8: 0.8045 Epoch 5/20 37/37 [==============================] - ETA: 0s - loss: 0.5477 - accuracy: 0.7294 - auc_8: 0.8099 Epoch 5: val_loss improved from 0.55460 to 0.53902, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 44s 1s/step - loss: 0.5477 - accuracy: 0.7294 - auc_8: 0.8099 - val_loss: 0.5390 - val_accuracy: 0.7335 - val_auc_8: 0.8209 Epoch 6/20 37/37 [==============================] - ETA: 0s - loss: 0.5301 - accuracy: 0.7440 - auc_8: 0.8280 Epoch 6: val_loss improved from 0.53902 to 0.52073, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 44s 1s/step - loss: 0.5301 - accuracy: 0.7440 - auc_8: 0.8280 - val_loss: 0.5207 - val_accuracy: 0.7582 - val_auc_8: 0.8387 Epoch 7/20 37/37 [==============================] - ETA: 0s - loss: 0.5032 - accuracy: 0.7632 - auc_8: 0.8516 Epoch 7: val_loss improved from 0.52073 to 0.48963, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 43s 1s/step - loss: 0.5032 - accuracy: 0.7632 - auc_8: 0.8516 - val_loss: 0.4896 - val_accuracy: 0.7891 - val_auc_8: 0.8652 Epoch 8/20 37/37 [==============================] - ETA: 0s - loss: 0.4616 - accuracy: 0.7957 - auc_8: 0.8809 Epoch 8: val_loss improved from 0.48963 to 0.43302, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 42s 1s/step - loss: 0.4616 - accuracy: 0.7957 - auc_8: 0.8809 - val_loss: 0.4330 - val_accuracy: 0.7987 - val_auc_8: 0.9011 Epoch 9/20 37/37 [==============================] - ETA: 0s - loss: 0.4018 - accuracy: 0.8295 - auc_8: 0.9130 Epoch 9: val_loss improved from 0.43302 to 0.36535, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 44s 1s/step - loss: 0.4018 - accuracy: 0.8295 - auc_8: 0.9130 - val_loss: 0.3653 - val_accuracy: 0.8482 - val_auc_8: 0.9269 Epoch 10/20 37/37 [==============================] - ETA: 0s - loss: 0.3333 - accuracy: 0.8641 - auc_8: 0.9433 Epoch 10: val_loss improved from 0.36535 to 0.30654, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 43s 1s/step - loss: 0.3333 - accuracy: 0.8641 - auc_8: 0.9433 - val_loss: 0.3065 - val_accuracy: 0.8753 - val_auc_8: 0.9537 Epoch 11/20 37/37 [==============================] - ETA: 0s - loss: 0.2829 - accuracy: 0.8930 - auc_8: 0.9592 Epoch 11: val_loss improved from 0.30654 to 0.25850, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 44s 1s/step - loss: 0.2829 - accuracy: 0.8930 - auc_8: 0.9592 - val_loss: 0.2585 - val_accuracy: 0.9040 - val_auc_8: 0.9647 Epoch 12/20 37/37 [==============================] - ETA: 0s - loss: 0.2439 - accuracy: 0.9139 - auc_8: 0.9684 Epoch 12: val_loss improved from 0.25850 to 0.23328, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 40s 1s/step - loss: 0.2439 - accuracy: 0.9139 - auc_8: 0.9684 - val_loss: 0.2333 - val_accuracy: 0.9179 - val_auc_8: 0.9705 Epoch 13/20 37/37 [==============================] - ETA: 0s - loss: 0.2165 - accuracy: 0.9247 - auc_8: 0.9740 Epoch 13: val_loss improved from 0.23328 to 0.21382, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 37s 1s/step - loss: 0.2165 - accuracy: 0.9247 - auc_8: 0.9740 - val_loss: 0.2138 - val_accuracy: 0.9204 - val_auc_8: 0.9735 Epoch 14/20 37/37 [==============================] - ETA: 0s - loss: 0.2007 - accuracy: 0.9323 - auc_8: 0.9768 Epoch 14: val_loss improved from 0.21382 to 0.19934, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 38s 1s/step - loss: 0.2007 - accuracy: 0.9323 - auc_8: 0.9768 - val_loss: 0.1993 - val_accuracy: 0.9272 - val_auc_8: 0.9758 Epoch 15/20 37/37 [==============================] - ETA: 0s - loss: 0.1879 - accuracy: 0.9361 - auc_8: 0.9786 Epoch 15: val_loss improved from 0.19934 to 0.19115, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 40s 1s/step - loss: 0.1879 - accuracy: 0.9361 - auc_8: 0.9786 - val_loss: 0.1912 - val_accuracy: 0.9301 - val_auc_8: 0.9769 Epoch 16/20 37/37 [==============================] - ETA: 0s - loss: 0.1755 - accuracy: 0.9393 - auc_8: 0.9816 Epoch 16: val_loss improved from 0.19115 to 0.18210, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 43s 1s/step - loss: 0.1755 - accuracy: 0.9393 - auc_8: 0.9816 - val_loss: 0.1821 - val_accuracy: 0.9396 - val_auc_8: 0.9799 Epoch 17/20 37/37 [==============================] - ETA: 0s - loss: 0.1679 - accuracy: 0.9423 - auc_8: 0.9827 Epoch 17: val_loss improved from 0.18210 to 0.17431, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 41s 1s/step - loss: 0.1679 - accuracy: 0.9423 - auc_8: 0.9827 - val_loss: 0.1743 - val_accuracy: 0.9409 - val_auc_8: 0.9814 Epoch 18/20 37/37 [==============================] - ETA: 0s - loss: 0.1575 - accuracy: 0.9474 - auc_8: 0.9845 Epoch 18: val_loss improved from 0.17431 to 0.16642, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 37s 992ms/step - loss: 0.1575 - accuracy: 0.9474 - auc_8: 0.9845 - val_loss: 0.1664 - val_accuracy: 0.9407 - val_auc_8: 0.9817 Epoch 19/20 37/37 [==============================] - ETA: 0s - loss: 0.1506 - accuracy: 0.9476 - auc_8: 0.9856 Epoch 19: val_loss improved from 0.16642 to 0.16199, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 39s 1s/step - loss: 0.1506 - accuracy: 0.9476 - auc_8: 0.9856 - val_loss: 0.1620 - val_accuracy: 0.9436 - val_auc_8: 0.9818 Epoch 20/20 37/37 [==============================] - ETA: 0s - loss: 0.1425 - accuracy: 0.9513 - auc_8: 0.9871 Epoch 20: val_loss improved from 0.16199 to 0.16068, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_1.keras 37/37 [==============================] - 39s 1s/step - loss: 0.1425 - accuracy: 0.9513 - auc_8: 0.9871 - val_loss: 0.1607 - val_accuracy: 0.9401 - val_auc_8: 0.9836 Training history saved to /content/drive/MyDrive/projects/school/Capstone Project/model_1_history.json
<keras.src.callbacks.History at 0x7bd2c08d5690>
model_1_predictions = predictions("model_1", test_images_64x64, test_labels_64x64)
Test accuracy: 0.9438461661338806 82/82 [==============================] - 2s 23ms/step
Plotting the confusion matrix
show_confusion(test_labels_64x64, model_1_predictions)
Plotting the train and validation curves
plot_accuracy("model_1")
So now let's try to build another model with few more add on layers and try to check if we can try to improve the model. Therefore try to build a model by adding few layers if required and altering the activation functions.
It's not the way I'd go, first very simple CNN has really good initial accuracy, I should tweak it, but as required for passing the exam purposes I'll do something more complex.
model_2 = Sequential([
Conv2D(18, kernel_size=(3, 3), padding='same', input_shape=(64, 64, 3)),
LeakyReLU(alpha=0.25),
Conv2D(16, kernel_size=(3, 3), padding='same'),
LeakyReLU(alpha=0.20),
Conv2D(12, kernel_size=(3, 3), padding='same'),
LeakyReLU(alpha=0.15),
MaxPooling2D(pool_size=(2, 2)),
Conv2D(8, kernel_size=(3, 3), padding='same'),
LeakyReLU(alpha=0.10),
Flatten(),
Dense(32, activation='sigmoid'),
Dense(18, activation='sigmoid'),
Dense(1, activation='sigmoid')
])
# Function defined at the top to not repeat the same code again and agian (DRY principle)
Using Callbacks
# Function defined at the top to not repeat the same code again and agian (DRY principle)
Fit and Train the model
compile_and_fit_model(model_2, "model_2",
images=train_images_64x64,
labels=train_labels_64x64,
validation_split=0.25,
epochs=25,
batch_size=512)
Epoch 1/25 37/37 [==============================] - ETA: 0s - loss: 0.6589 - accuracy: 0.6058 - auc_9: 0.6595 Epoch 1: val_loss improved from inf to 0.62423, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 235s 6s/step - loss: 0.6589 - accuracy: 0.6058 - auc_9: 0.6595 - val_loss: 0.6242 - val_accuracy: 0.6726 - val_auc_9: 0.7289 Epoch 2/25 37/37 [==============================] - ETA: 0s - loss: 0.6009 - accuracy: 0.6954 - auc_9: 0.7510 Epoch 2: val_loss improved from 0.62423 to 0.58834, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 194s 5s/step - loss: 0.6009 - accuracy: 0.6954 - auc_9: 0.7510 - val_loss: 0.5883 - val_accuracy: 0.6938 - val_auc_9: 0.7672 Epoch 3/25 37/37 [==============================] - ETA: 0s - loss: 0.5651 - accuracy: 0.7164 - auc_9: 0.7882 Epoch 3: val_loss improved from 0.58834 to 0.57091, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 214s 6s/step - loss: 0.5651 - accuracy: 0.7164 - auc_9: 0.7882 - val_loss: 0.5709 - val_accuracy: 0.7167 - val_auc_9: 0.7898 Epoch 4/25 37/37 [==============================] - ETA: 0s - loss: 0.5433 - accuracy: 0.7288 - auc_9: 0.8032 Epoch 4: val_loss improved from 0.57091 to 0.52368, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 194s 5s/step - loss: 0.5433 - accuracy: 0.7288 - auc_9: 0.8032 - val_loss: 0.5237 - val_accuracy: 0.7415 - val_auc_9: 0.8242 Epoch 5/25 37/37 [==============================] - ETA: 0s - loss: 0.4936 - accuracy: 0.7713 - auc_9: 0.8432 Epoch 5: val_loss improved from 0.52368 to 0.45754, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 191s 5s/step - loss: 0.4936 - accuracy: 0.7713 - auc_9: 0.8432 - val_loss: 0.4575 - val_accuracy: 0.8016 - val_auc_9: 0.8716 Epoch 6/25 37/37 [==============================] - ETA: 0s - loss: 0.3936 - accuracy: 0.8553 - auc_9: 0.9057 Epoch 6: val_loss improved from 0.45754 to 0.31239, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 193s 5s/step - loss: 0.3936 - accuracy: 0.8553 - auc_9: 0.9057 - val_loss: 0.3124 - val_accuracy: 0.9018 - val_auc_9: 0.9362 Epoch 7/25 37/37 [==============================] - ETA: 0s - loss: 0.2834 - accuracy: 0.9120 - auc_9: 0.9405 Epoch 7: val_loss improved from 0.31239 to 0.25873, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 194s 5s/step - loss: 0.2834 - accuracy: 0.9120 - auc_9: 0.9405 - val_loss: 0.2587 - val_accuracy: 0.9231 - val_auc_9: 0.9493 Epoch 8/25 37/37 [==============================] - ETA: 0s - loss: 0.2467 - accuracy: 0.9264 - auc_9: 0.9501 Epoch 8: val_loss improved from 0.25873 to 0.23320, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 189s 5s/step - loss: 0.2467 - accuracy: 0.9264 - auc_9: 0.9501 - val_loss: 0.2332 - val_accuracy: 0.9333 - val_auc_9: 0.9550 Epoch 9/25 37/37 [==============================] - ETA: 0s - loss: 0.2304 - accuracy: 0.9329 - auc_9: 0.9555 Epoch 9: val_loss improved from 0.23320 to 0.21894, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 193s 5s/step - loss: 0.2304 - accuracy: 0.9329 - auc_9: 0.9555 - val_loss: 0.2189 - val_accuracy: 0.9369 - val_auc_9: 0.9623 Epoch 10/25 37/37 [==============================] - ETA: 0s - loss: 0.2177 - accuracy: 0.9365 - auc_9: 0.9594 Epoch 10: val_loss improved from 0.21894 to 0.21057, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 194s 5s/step - loss: 0.2177 - accuracy: 0.9365 - auc_9: 0.9594 - val_loss: 0.2106 - val_accuracy: 0.9377 - val_auc_9: 0.9662 Epoch 11/25 37/37 [==============================] - ETA: 0s - loss: 0.2055 - accuracy: 0.9395 - auc_9: 0.9660 Epoch 11: val_loss improved from 0.21057 to 0.19941, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 193s 5s/step - loss: 0.2055 - accuracy: 0.9395 - auc_9: 0.9660 - val_loss: 0.1994 - val_accuracy: 0.9418 - val_auc_9: 0.9705 Epoch 12/25 37/37 [==============================] - ETA: 0s - loss: 0.1930 - accuracy: 0.9445 - auc_9: 0.9717 Epoch 12: val_loss improved from 0.19941 to 0.19403, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 192s 5s/step - loss: 0.1930 - accuracy: 0.9445 - auc_9: 0.9717 - val_loss: 0.1940 - val_accuracy: 0.9399 - val_auc_9: 0.9741 Epoch 13/25 37/37 [==============================] - ETA: 0s - loss: 0.1794 - accuracy: 0.9472 - auc_9: 0.9774 Epoch 13: val_loss improved from 0.19403 to 0.17134, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 192s 5s/step - loss: 0.1794 - accuracy: 0.9472 - auc_9: 0.9774 - val_loss: 0.1713 - val_accuracy: 0.9498 - val_auc_9: 0.9784 Epoch 14/25 37/37 [==============================] - ETA: 0s - loss: 0.1674 - accuracy: 0.9491 - auc_9: 0.9796 Epoch 14: val_loss did not improve from 0.17134 37/37 [==============================] - 192s 5s/step - loss: 0.1674 - accuracy: 0.9491 - auc_9: 0.9796 - val_loss: 0.2078 - val_accuracy: 0.9449 - val_auc_9: 0.9812 Epoch 15/25 37/37 [==============================] - ETA: 0s - loss: 0.1604 - accuracy: 0.9513 - auc_9: 0.9805 Epoch 15: val_loss improved from 0.17134 to 0.14759, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 193s 5s/step - loss: 0.1604 - accuracy: 0.9513 - auc_9: 0.9805 - val_loss: 0.1476 - val_accuracy: 0.9524 - val_auc_9: 0.9879 Epoch 16/25 37/37 [==============================] - ETA: 0s - loss: 0.1369 - accuracy: 0.9595 - auc_9: 0.9860 Epoch 16: val_loss did not improve from 0.14759 37/37 [==============================] - 193s 5s/step - loss: 0.1369 - accuracy: 0.9595 - auc_9: 0.9860 - val_loss: 0.1649 - val_accuracy: 0.9522 - val_auc_9: 0.9886 Epoch 17/25 37/37 [==============================] - ETA: 0s - loss: 0.1339 - accuracy: 0.9589 - auc_9: 0.9848 Epoch 17: val_loss improved from 0.14759 to 0.13039, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 193s 5s/step - loss: 0.1339 - accuracy: 0.9589 - auc_9: 0.9848 - val_loss: 0.1304 - val_accuracy: 0.9588 - val_auc_9: 0.9902 Epoch 18/25 37/37 [==============================] - ETA: 0s - loss: 0.1140 - accuracy: 0.9659 - auc_9: 0.9894 Epoch 18: val_loss improved from 0.13039 to 0.11588, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 193s 5s/step - loss: 0.1140 - accuracy: 0.9659 - auc_9: 0.9894 - val_loss: 0.1159 - val_accuracy: 0.9657 - val_auc_9: 0.9897 Epoch 19/25 37/37 [==============================] - ETA: 0s - loss: 0.1056 - accuracy: 0.9689 - auc_9: 0.9899 Epoch 19: val_loss improved from 0.11588 to 0.10842, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 193s 5s/step - loss: 0.1056 - accuracy: 0.9689 - auc_9: 0.9899 - val_loss: 0.1084 - val_accuracy: 0.9696 - val_auc_9: 0.9913 Epoch 20/25 37/37 [==============================] - ETA: 0s - loss: 0.1011 - accuracy: 0.9705 - auc_9: 0.9903 Epoch 20: val_loss improved from 0.10842 to 0.10509, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 193s 5s/step - loss: 0.1011 - accuracy: 0.9705 - auc_9: 0.9903 - val_loss: 0.1051 - val_accuracy: 0.9697 - val_auc_9: 0.9913 Epoch 21/25 37/37 [==============================] - ETA: 0s - loss: 0.0962 - accuracy: 0.9713 - auc_9: 0.9920 Epoch 21: val_loss did not improve from 0.10509 37/37 [==============================] - 192s 5s/step - loss: 0.0962 - accuracy: 0.9713 - auc_9: 0.9920 - val_loss: 0.1057 - val_accuracy: 0.9678 - val_auc_9: 0.9912 Epoch 22/25 37/37 [==============================] - ETA: 0s - loss: 0.0880 - accuracy: 0.9740 - auc_9: 0.9932 Epoch 22: val_loss improved from 0.10509 to 0.09637, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_2.keras 37/37 [==============================] - 193s 5s/step - loss: 0.0880 - accuracy: 0.9740 - auc_9: 0.9932 - val_loss: 0.0964 - val_accuracy: 0.9702 - val_auc_9: 0.9926 Epoch 23/25 37/37 [==============================] - ETA: 0s - loss: 0.0834 - accuracy: 0.9759 - auc_9: 0.9932 Epoch 23: val_loss did not improve from 0.09637 37/37 [==============================] - 189s 5s/step - loss: 0.0834 - accuracy: 0.9759 - auc_9: 0.9932 - val_loss: 0.1068 - val_accuracy: 0.9702 - val_auc_9: 0.9918 Epoch 24/25 37/37 [==============================] - ETA: 0s - loss: 0.0816 - accuracy: 0.9758 - auc_9: 0.9942 Epoch 24: val_loss did not improve from 0.09637 37/37 [==============================] - 191s 5s/step - loss: 0.0816 - accuracy: 0.9758 - auc_9: 0.9942 - val_loss: 0.0965 - val_accuracy: 0.9715 - val_auc_9: 0.9922 Epoch 25/25 37/37 [==============================] - ETA: 0s - loss: 0.0802 - accuracy: 0.9762 - auc_9: 0.9939 Epoch 25: val_loss did not improve from 0.09637 37/37 [==============================] - 192s 5s/step - loss: 0.0802 - accuracy: 0.9762 - auc_9: 0.9939 - val_loss: 0.1087 - val_accuracy: 0.9660 - val_auc_9: 0.9917 Training history saved to /content/drive/MyDrive/projects/school/Capstone Project/model_2_history.json
<keras.src.callbacks.History at 0x7bd2c04bee30>
model_2_predictions = predictions("model_2", test_images_64x64, test_labels_64x64)
Test accuracy: 0.9680769443511963 82/82 [==============================] - 8s 92ms/step
Plotting the confusion matrix
show_confusion(model_2_predictions, test_labels_64x64)
Plotting the train and the validation curves
plot_accuracy("model_2")
Now let's build a model with LeakyRelu as the activation function
Let us try to build a model using BatchNormalization and using LeakyRelu as our activation function.
NOTE: not sure how to read this instruction:
Trying to follow unclear and inconstent instructions I'll just use previous model and add some normalization.
model_3 = Sequential([
Conv2D(18, kernel_size=(3, 3), padding='same', input_shape=(64, 64, 3)),
LeakyReLU(alpha=0.25),
BatchNormalization(),
Conv2D(16, kernel_size=(3, 3), padding='same'),
LeakyReLU(alpha=0.20),
BatchNormalization(),
Conv2D(12, kernel_size=(3, 3), padding='same'),
LeakyReLU(alpha=0.15),
MaxPooling2D(pool_size=(2, 2)),
Conv2D(8, kernel_size=(3, 3), padding='same'),
LeakyReLU(alpha=0.10),
Flatten(),
Dense(32, activation='sigmoid'),
BatchNormalization(),
Dense(18, activation='sigmoid'),
Dense(1, activation='sigmoid')
])
# Function defined at the top to not repeat the same code again and agian (DRY principle)
Using callbacks
# Function defined at the top to not repeat the same code again and agian (DRY principle)
Fit and train the model
compile_and_fit_model(model_3, "model_3",
images=train_images_64x64,
labels=train_labels_64x64,
epochs=13)
Epoch 1/13 37/37 [==============================] - ETA: 0s - loss: 0.5218 - accuracy: 0.7450 - auc_10: 0.8167 Epoch 1: val_loss improved from inf to 0.61325, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_3.keras 37/37 [==============================] - 253s 7s/step - loss: 0.5218 - accuracy: 0.7450 - auc_10: 0.8167 - val_loss: 0.6133 - val_accuracy: 0.7470 - val_auc_10: 0.7958 Epoch 2/13 37/37 [==============================] - ETA: 0s - loss: 0.3189 - accuracy: 0.8915 - auc_10: 0.9420 Epoch 2: val_loss did not improve from 0.61325 37/37 [==============================] - 238s 6s/step - loss: 0.3189 - accuracy: 0.8915 - auc_10: 0.9420 - val_loss: 0.6232 - val_accuracy: 0.7317 - val_auc_10: 0.8637 Epoch 3/13 37/37 [==============================] - ETA: 0s - loss: 0.2002 - accuracy: 0.9479 - auc_10: 0.9820 Epoch 3: val_loss did not improve from 0.61325 37/37 [==============================] - 260s 7s/step - loss: 0.2002 - accuracy: 0.9479 - auc_10: 0.9820 - val_loss: 0.7093 - val_accuracy: 0.5208 - val_auc_10: 0.7837 Epoch 4/13 37/37 [==============================] - ETA: 0s - loss: 0.1511 - accuracy: 0.9624 - auc_10: 0.9896 Epoch 4: val_loss did not improve from 0.61325 37/37 [==============================] - 257s 7s/step - loss: 0.1511 - accuracy: 0.9624 - auc_10: 0.9896 - val_loss: 0.8944 - val_accuracy: 0.5051 - val_auc_10: 0.8030 Epoch 5/13 37/37 [==============================] - ETA: 0s - loss: 0.1141 - accuracy: 0.9737 - auc_10: 0.9942 Epoch 5: val_loss did not improve from 0.61325 37/37 [==============================] - 258s 7s/step - loss: 0.1141 - accuracy: 0.9737 - auc_10: 0.9942 - val_loss: 0.8273 - val_accuracy: 0.5498 - val_auc_10: 0.8284 Epoch 6/13 37/37 [==============================] - ETA: 0s - loss: 0.0887 - accuracy: 0.9818 - auc_10: 0.9962 Epoch 6: val_loss did not improve from 0.61325 37/37 [==============================] - 239s 6s/step - loss: 0.0887 - accuracy: 0.9818 - auc_10: 0.9962 - val_loss: 0.9999 - val_accuracy: 0.5691 - val_auc_10: 0.8751 Epoch 7/13 37/37 [==============================] - ETA: 0s - loss: 0.0750 - accuracy: 0.9844 - auc_10: 0.9969 Epoch 7: val_loss did not improve from 0.61325 37/37 [==============================] - 257s 7s/step - loss: 0.0750 - accuracy: 0.9844 - auc_10: 0.9969 - val_loss: 1.2973 - val_accuracy: 0.5662 - val_auc_10: 0.8833 Epoch 8/13 37/37 [==============================] - ETA: 0s - loss: 0.0612 - accuracy: 0.9887 - auc_10: 0.9974 Epoch 8: val_loss did not improve from 0.61325 37/37 [==============================] - 259s 7s/step - loss: 0.0612 - accuracy: 0.9887 - auc_10: 0.9974 - val_loss: 1.0363 - val_accuracy: 0.6726 - val_auc_10: 0.9022 Epoch 9/13 37/37 [==============================] - ETA: 0s - loss: 0.0507 - accuracy: 0.9916 - auc_10: 0.9979 Epoch 9: val_loss did not improve from 0.61325 37/37 [==============================] - 259s 7s/step - loss: 0.0507 - accuracy: 0.9916 - auc_10: 0.9979 - val_loss: 0.6566 - val_accuracy: 0.7929 - val_auc_10: 0.9523 Epoch 10/13 37/37 [==============================] - ETA: 0s - loss: 0.0425 - accuracy: 0.9935 - auc_10: 0.9985 Epoch 10: val_loss improved from 0.61325 to 0.49473, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_3.keras 37/37 [==============================] - 239s 6s/step - loss: 0.0425 - accuracy: 0.9935 - auc_10: 0.9985 - val_loss: 0.4947 - val_accuracy: 0.8558 - val_auc_10: 0.9503 Epoch 11/13 37/37 [==============================] - ETA: 0s - loss: 0.0361 - accuracy: 0.9952 - auc_10: 0.9986 Epoch 11: val_loss improved from 0.49473 to 0.36257, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_3.keras 37/37 [==============================] - 236s 6s/step - loss: 0.0361 - accuracy: 0.9952 - auc_10: 0.9986 - val_loss: 0.3626 - val_accuracy: 0.9006 - val_auc_10: 0.9717 Epoch 12/13 37/37 [==============================] - ETA: 0s - loss: 0.0344 - accuracy: 0.9943 - auc_10: 0.9987 Epoch 12: val_loss improved from 0.36257 to 0.36255, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_3.keras 37/37 [==============================] - 258s 7s/step - loss: 0.0344 - accuracy: 0.9943 - auc_10: 0.9987 - val_loss: 0.3626 - val_accuracy: 0.9046 - val_auc_10: 0.9590 Epoch 13/13 37/37 [==============================] - ETA: 0s - loss: 0.0292 - accuracy: 0.9956 - auc_10: 0.9990 Epoch 13: val_loss improved from 0.36255 to 0.26232, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_3.keras 37/37 [==============================] - 239s 6s/step - loss: 0.0292 - accuracy: 0.9956 - auc_10: 0.9990 - val_loss: 0.2623 - val_accuracy: 0.9364 - val_auc_10: 0.9680 Training history saved to /content/drive/MyDrive/projects/school/Capstone Project/model_3_history.json
<keras.src.callbacks.History at 0x7bd2be18cfd0>
Plotting the train and validation accuracy
plot_accuracy("model_3")
model_3_predictions = predictions("model_3", test_images_64x64, test_labels_64x64)
Test accuracy: 0.9411538243293762 82/82 [==============================] - 8s 91ms/step
It was great lesson, as I runned this twice, because once file did not save and I couldn't load model to make predictions and shape of validation accuracy was very different. Anyway - because such high difference between training accuracy and validation accuracy this model shouldn't be trusted, I wouldn't use anything like this without much wider tests. So BatchNormalization can make model perform worse.
Generate the classification report and confusion matrix
print("Classification Report:")
print(classification_report(test_labels_64x64, model_3_predictions))
Classification Report:
precision recall f1-score support
0 0.90 1.00 0.94 1300
1 1.00 0.89 0.94 1300
accuracy 0.94 2600
macro avg 0.95 0.94 0.94 2600
weighted avg 0.95 0.94 0.94 2600
show_confusion(model_3_predictions, test_labels_64x64)
I don't believe data augumentation will do much good. Data set is relatively large, especially if we strive for small model that can work with number of cells in range of dozen hunders per single test sample. Additionally data is quite divers. Cells in training data are in various ratios, rotations, and from few microspocopes with various lighning conditions... however this last point where we could still try to do better: Color Jittering could be applied.
Since model_2 had accuracy of 0.98 without even tweaking hyperparameters, and it's definitely not the production model there is very little reason to try next model instead of optimizing previous ones. So I'll do something little different - I'll generate copy of dataset with cell size of 8x8 pixels what more closely resembles what we would need for production model that can be wiedly adapted. I won't go for detailed analysis, just run previous model to have insight how problematic and how hard it may be to optimize model for production. What works in well establisched lab with carefully prepared samples by very well skilled technician using state of the art aparature may not be (even with lot of work) applicable to production where smaples will be much lower quality and model has to work has to be efficient.
Image Data Generator is depreciated and not advised, so I'll do it other way - especially that above I presented how to preprocess data in separate process.
# I've chosen different startegy, to have preprocessing directly in the model,
# while training data is rescaled in a file for easy loading, but this is how it would be done.
# Additionally ImageDataGenerator is depreciated: https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator
test_datagen = ImageDataGenerator(rescale = 1.0/255.)
validation_generator = test_datagen.flow_from_directory(
test_data_folder,
batch_size=20,
class_mode = 'binary',
target_size = (8, 8))
train_datagen = ImageDataGenerator(
brightness_range=(0.7, 1.0),
channel_shift_range=0.2,
horizontal_flip = True,
vertical_flip = True)
train_generator = train_datagen.flow_from_directory(
train_data_folder,
batch_size=20,
class_mode='binary',
target_size=(8, 8))
Found 2600 images belonging to 2 classes. Found 24958 images belonging to 2 classes.
# Code below could be clearer, but then I'd need to rerun whole notebook
# And given restrictions applied by googl on colab it may take too long
@register_keras_serializable(package='Custom', name='RgbToHsv')
class RgbToHsv(tf.keras.layers.Layer):
def __init__(self):
super(RgbToHsv, self).__init__()
def call(self, inputs):
return tf.image.rgb_to_hsv(inputs)
@register_keras_serializable(package='Custom', name='HsvToRgb')
class HsvToRgb(tf.keras.layers.Layer):
def __init__(self):
super(HsvToRgb, self).__init__()
def call(self, inputs):
return tf.image.hsv_to_rgb(inputs)
@register_keras_serializable(package='Custom', name='ColorShift')
class ColorShift(tf.keras.layers.Layer):
def __init__(self, factor=0.1):
super(ColorShift, self).__init__()
self.factor = factor
def call(self, inputs, training=None):
if not training:
return inputs
# Randomly adjust the hue of the input images
delta = tf.random.uniform((), -self.factor, self.factor)
return tf.image.adjust_hue(inputs, delta)
# I'm aware there is keras.RandomBrigness(...)
# but it was causing some troubles, so easier to simply redefine...
# later I should dug in and understand why it behaves so unexpectedly
@register_keras_serializable(package='Custom', name='BrightnessShift')
class BrightnessShift(tf.keras.layers.Layer):
def __init__(self, factor=0.2):
super(BrightnessShift, self).__init__()
self.factor = factor
def call(self, inputs):
adjusted = inputs + tf.random.uniform(shape=[], minval=-self.factor, maxval=self.factor)
return tf.clip_by_value(adjusted, clip_value_min=0.0, clip_value_max=1.0)
# Color shifting and brightness randomization require rgb color space, therefore we
# needed to split preprocessing into two stages - one before randomization, and second one after
early_preprocessing = tf.keras.Sequential([
HsvToRgb(), # normally what wouldn't be required but that's my preprocessed data
#tf.keras.layers.Rescaling(1./255)
])
# Data is really well diverse in shapes, sizes, rotations, etc.
# One thing that still could be more diferese is to simulate more different
# Ligting conditions
trainig_data_augmentation = tf.keras.Sequential([
BrightnessShift(factor=0.2),
tf.keras.layers.RandomContrast(factor=0.2),
ColorShift(factor=0.2),
])
# It's late, as converting to HSV color space must be done after working with colors
# Color functions assume they work on RGB data
late_preprocessing = tf.keras.Sequential([
RgbToHsv()
])
I might run it with previous data on different models, but I don't want this notebook to explode into long book. And it's always nice to stick to chosen goal instead of doing fancy stuff to show off.
augmented_images = test_images_64x64[:32]
augmented_images = early_preprocessing(tf.stack(augmented_images, axis=0), training=True)
augmented_images = trainig_data_augmentation(augmented_images, training=True)
augmented_images_labels = test_labels_64x64[:32]
ig, axes = plt.subplots(nrows=4, ncols=8, figsize=(10, 7))
for i, ax in enumerate(axes.flat):
ax.imshow(augmented_images[i])
ax.axis('off')
if augmented_images_labels[i] == 0:
label = healthy
else:
label = infected
ax.text(0, -2, label, fontsize=8, ha='left')
plt.tight_layout()
plt.show()
For comparision reasons I'm using best model so far. Difference is that data preprocessing is included in model, and we use data augmentation for training and finally image resolution is much lower.
model_4 = Sequential([
early_preprocessing,
trainig_data_augmentation,
late_preprocessing,
Conv2D(18, kernel_size=(3, 3), padding='same', input_shape=(64, 64, 3)),
LeakyReLU(alpha=0.25),
Conv2D(16, kernel_size=(3, 3), padding='same'),
LeakyReLU(alpha=0.20),
Conv2D(12, kernel_size=(3, 3), padding='same'),
LeakyReLU(alpha=0.15),
MaxPooling2D(pool_size=(2, 2)),
Conv2D(8, kernel_size=(3, 3), padding='same'),
LeakyReLU(alpha=0.10),
Flatten(),
Dense(64, activation='sigmoid'),
Dense(16, activation='sigmoid'),
Dense(1, activation='sigmoid')
])
Using Callbacks
# inside function compile_and_fit_model (DRY principle)
Fit and Train the model
compile_and_fit_model(model_4, "model_4", train_images_64x64, train_labels_64x64, epochs=50, learning_rate=0.0002)
Epoch 1/50 37/37 [==============================] - ETA: 0s - loss: 0.6763 - accuracy: 0.5837 - auc: 0.6150 Epoch 1: val_loss improved from inf to 0.65130, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 22s 273ms/step - loss: 0.6763 - accuracy: 0.5837 - auc: 0.6150 - val_loss: 0.6513 - val_accuracy: 0.6256 - val_auc: 0.6836 Epoch 2/50 37/37 [==============================] - ETA: 0s - loss: 0.6622 - accuracy: 0.6107 - auc: 0.6412 Epoch 2: val_loss improved from 0.65130 to 0.64230, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 109ms/step - loss: 0.6622 - accuracy: 0.6107 - auc: 0.6412 - val_loss: 0.6423 - val_accuracy: 0.6353 - val_auc: 0.7007 Epoch 3/50 37/37 [==============================] - ETA: 0s - loss: 0.6518 - accuracy: 0.6243 - auc: 0.6657 Epoch 3: val_loss improved from 0.64230 to 0.63821, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 5s 126ms/step - loss: 0.6518 - accuracy: 0.6243 - auc: 0.6657 - val_loss: 0.6382 - val_accuracy: 0.6399 - val_auc: 0.7165 Epoch 4/50 37/37 [==============================] - ETA: 0s - loss: 0.6414 - accuracy: 0.6444 - auc: 0.6855 Epoch 4: val_loss improved from 0.63821 to 0.63065, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 112ms/step - loss: 0.6414 - accuracy: 0.6444 - auc: 0.6855 - val_loss: 0.6306 - val_accuracy: 0.6564 - val_auc: 0.7158 Epoch 5/50 37/37 [==============================] - ETA: 0s - loss: 0.6407 - accuracy: 0.6352 - auc: 0.6863 Epoch 5: val_loss improved from 0.63065 to 0.62376, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 5s 124ms/step - loss: 0.6407 - accuracy: 0.6352 - auc: 0.6863 - val_loss: 0.6238 - val_accuracy: 0.6663 - val_auc: 0.7346 Epoch 6/50 37/37 [==============================] - ETA: 0s - loss: 0.6335 - accuracy: 0.6484 - auc: 0.7019 Epoch 6: val_loss improved from 0.62376 to 0.61499, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 121ms/step - loss: 0.6335 - accuracy: 0.6484 - auc: 0.7019 - val_loss: 0.6150 - val_accuracy: 0.6724 - val_auc: 0.7414 Epoch 7/50 37/37 [==============================] - ETA: 0s - loss: 0.6318 - accuracy: 0.6475 - auc: 0.7009 Epoch 7: val_loss did not improve from 0.61499 37/37 [==============================] - 4s 108ms/step - loss: 0.6318 - accuracy: 0.6475 - auc: 0.7009 - val_loss: 0.6702 - val_accuracy: 0.5893 - val_auc: 0.7259 Epoch 8/50 37/37 [==============================] - ETA: 0s - loss: 0.6293 - accuracy: 0.6511 - auc: 0.7072 Epoch 8: val_loss improved from 0.61499 to 0.60813, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 116ms/step - loss: 0.6293 - accuracy: 0.6511 - auc: 0.7072 - val_loss: 0.6081 - val_accuracy: 0.6843 - val_auc: 0.7579 Epoch 9/50 37/37 [==============================] - ETA: 0s - loss: 0.6120 - accuracy: 0.6760 - auc: 0.7362 Epoch 9: val_loss improved from 0.60813 to 0.60550, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 115ms/step - loss: 0.6120 - accuracy: 0.6760 - auc: 0.7362 - val_loss: 0.6055 - val_accuracy: 0.6809 - val_auc: 0.7632 Epoch 10/50 37/37 [==============================] - ETA: 0s - loss: 0.6098 - accuracy: 0.6815 - auc: 0.7418 Epoch 10: val_loss improved from 0.60550 to 0.59206, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 118ms/step - loss: 0.6098 - accuracy: 0.6815 - auc: 0.7418 - val_loss: 0.5921 - val_accuracy: 0.6947 - val_auc: 0.7762 Epoch 11/50 37/37 [==============================] - ETA: 0s - loss: 0.6083 - accuracy: 0.6823 - auc: 0.7363 Epoch 11: val_loss improved from 0.59206 to 0.58370, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 116ms/step - loss: 0.6083 - accuracy: 0.6823 - auc: 0.7363 - val_loss: 0.5837 - val_accuracy: 0.7074 - val_auc: 0.7830 Epoch 12/50 37/37 [==============================] - ETA: 0s - loss: 0.5993 - accuracy: 0.6951 - auc: 0.7502 Epoch 12: val_loss improved from 0.58370 to 0.57271, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 117ms/step - loss: 0.5993 - accuracy: 0.6951 - auc: 0.7502 - val_loss: 0.5727 - val_accuracy: 0.7184 - val_auc: 0.7931 Epoch 13/50 37/37 [==============================] - ETA: 0s - loss: 0.5957 - accuracy: 0.6914 - auc: 0.7560 Epoch 13: val_loss did not improve from 0.57271 37/37 [==============================] - 4s 108ms/step - loss: 0.5957 - accuracy: 0.6914 - auc: 0.7560 - val_loss: 0.6108 - val_accuracy: 0.6575 - val_auc: 0.8121 Epoch 14/50 37/37 [==============================] - ETA: 0s - loss: 0.5815 - accuracy: 0.7036 - auc: 0.7734 Epoch 14: val_loss improved from 0.57271 to 0.55774, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 110ms/step - loss: 0.5815 - accuracy: 0.7036 - auc: 0.7734 - val_loss: 0.5577 - val_accuracy: 0.7287 - val_auc: 0.8092 Epoch 15/50 37/37 [==============================] - ETA: 0s - loss: 0.5753 - accuracy: 0.7139 - auc: 0.7823 Epoch 15: val_loss improved from 0.55774 to 0.53989, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 115ms/step - loss: 0.5753 - accuracy: 0.7139 - auc: 0.7823 - val_loss: 0.5399 - val_accuracy: 0.7389 - val_auc: 0.8334 Epoch 16/50 37/37 [==============================] - ETA: 0s - loss: 0.5654 - accuracy: 0.7248 - auc: 0.7948 Epoch 16: val_loss improved from 0.53989 to 0.53627, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 118ms/step - loss: 0.5654 - accuracy: 0.7248 - auc: 0.7948 - val_loss: 0.5363 - val_accuracy: 0.7375 - val_auc: 0.8386 Epoch 17/50 37/37 [==============================] - ETA: 0s - loss: 0.5481 - accuracy: 0.7359 - auc: 0.8093 Epoch 17: val_loss improved from 0.53627 to 0.51496, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 114ms/step - loss: 0.5481 - accuracy: 0.7359 - auc: 0.8093 - val_loss: 0.5150 - val_accuracy: 0.7587 - val_auc: 0.8516 Epoch 18/50 37/37 [==============================] - ETA: 0s - loss: 0.5431 - accuracy: 0.7430 - auc: 0.8171 Epoch 18: val_loss improved from 0.51496 to 0.49036, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 113ms/step - loss: 0.5431 - accuracy: 0.7430 - auc: 0.8171 - val_loss: 0.4904 - val_accuracy: 0.7821 - val_auc: 0.8680 Epoch 19/50 37/37 [==============================] - ETA: 0s - loss: 0.5292 - accuracy: 0.7516 - auc: 0.8247 Epoch 19: val_loss improved from 0.49036 to 0.48999, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 120ms/step - loss: 0.5292 - accuracy: 0.7516 - auc: 0.8247 - val_loss: 0.4900 - val_accuracy: 0.7989 - val_auc: 0.8765 Epoch 20/50 37/37 [==============================] - ETA: 0s - loss: 0.5084 - accuracy: 0.7723 - auc: 0.8449 Epoch 20: val_loss improved from 0.48999 to 0.46610, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 113ms/step - loss: 0.5084 - accuracy: 0.7723 - auc: 0.8449 - val_loss: 0.4661 - val_accuracy: 0.7984 - val_auc: 0.8756 Epoch 21/50 37/37 [==============================] - ETA: 0s - loss: 0.4873 - accuracy: 0.7826 - auc: 0.8615 Epoch 21: val_loss improved from 0.46610 to 0.45939, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 114ms/step - loss: 0.4873 - accuracy: 0.7826 - auc: 0.8615 - val_loss: 0.4594 - val_accuracy: 0.7971 - val_auc: 0.8907 Epoch 22/50 37/37 [==============================] - ETA: 0s - loss: 0.4643 - accuracy: 0.8041 - auc: 0.8771 Epoch 22: val_loss improved from 0.45939 to 0.42446, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 116ms/step - loss: 0.4643 - accuracy: 0.8041 - auc: 0.8771 - val_loss: 0.4245 - val_accuracy: 0.8236 - val_auc: 0.9017 Epoch 23/50 37/37 [==============================] - ETA: 0s - loss: 0.4532 - accuracy: 0.8090 - auc: 0.8793 Epoch 23: val_loss improved from 0.42446 to 0.40518, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 108ms/step - loss: 0.4532 - accuracy: 0.8090 - auc: 0.8793 - val_loss: 0.4052 - val_accuracy: 0.8359 - val_auc: 0.9058 Epoch 24/50 37/37 [==============================] - ETA: 0s - loss: 0.4362 - accuracy: 0.8226 - auc: 0.8905 Epoch 24: val_loss improved from 0.40518 to 0.37178, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 114ms/step - loss: 0.4362 - accuracy: 0.8226 - auc: 0.8905 - val_loss: 0.3718 - val_accuracy: 0.8625 - val_auc: 0.9311 Epoch 25/50 37/37 [==============================] - ETA: 0s - loss: 0.4252 - accuracy: 0.8289 - auc: 0.8960 Epoch 25: val_loss did not improve from 0.37178 37/37 [==============================] - 4s 114ms/step - loss: 0.4252 - accuracy: 0.8289 - auc: 0.8960 - val_loss: 0.3797 - val_accuracy: 0.8450 - val_auc: 0.9176 Epoch 26/50 37/37 [==============================] - ETA: 0s - loss: 0.3875 - accuracy: 0.8531 - auc: 0.9185 Epoch 26: val_loss improved from 0.37178 to 0.37118, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 110ms/step - loss: 0.3875 - accuracy: 0.8531 - auc: 0.9185 - val_loss: 0.3712 - val_accuracy: 0.8508 - val_auc: 0.9257 Epoch 27/50 37/37 [==============================] - ETA: 0s - loss: 0.3996 - accuracy: 0.8416 - auc: 0.9083 Epoch 27: val_loss improved from 0.37118 to 0.36531, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 115ms/step - loss: 0.3996 - accuracy: 0.8416 - auc: 0.9083 - val_loss: 0.3653 - val_accuracy: 0.8529 - val_auc: 0.9263 Epoch 28/50 37/37 [==============================] - ETA: 0s - loss: 0.3675 - accuracy: 0.8545 - auc: 0.9258 Epoch 28: val_loss improved from 0.36531 to 0.34906, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 118ms/step - loss: 0.3675 - accuracy: 0.8545 - auc: 0.9258 - val_loss: 0.3491 - val_accuracy: 0.8718 - val_auc: 0.9342 Epoch 29/50 37/37 [==============================] - ETA: 0s - loss: 0.3661 - accuracy: 0.8607 - auc: 0.9273 Epoch 29: val_loss improved from 0.34906 to 0.33116, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 111ms/step - loss: 0.3661 - accuracy: 0.8607 - auc: 0.9273 - val_loss: 0.3312 - val_accuracy: 0.8764 - val_auc: 0.9399 Epoch 30/50 37/37 [==============================] - ETA: 0s - loss: 0.3653 - accuracy: 0.8578 - auc: 0.9262 Epoch 30: val_loss did not improve from 0.33116 37/37 [==============================] - 4s 107ms/step - loss: 0.3653 - accuracy: 0.8578 - auc: 0.9262 - val_loss: 0.3764 - val_accuracy: 0.8460 - val_auc: 0.9302 Epoch 31/50 37/37 [==============================] - ETA: 0s - loss: 0.3494 - accuracy: 0.8696 - auc: 0.9309 Epoch 31: val_loss improved from 0.33116 to 0.30140, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 112ms/step - loss: 0.3494 - accuracy: 0.8696 - auc: 0.9309 - val_loss: 0.3014 - val_accuracy: 0.8896 - val_auc: 0.9550 Epoch 32/50 37/37 [==============================] - ETA: 0s - loss: 0.3237 - accuracy: 0.8785 - auc: 0.9444 Epoch 32: val_loss did not improve from 0.30140 37/37 [==============================] - 4s 116ms/step - loss: 0.3237 - accuracy: 0.8785 - auc: 0.9444 - val_loss: 0.3048 - val_accuracy: 0.8921 - val_auc: 0.9547 Epoch 33/50 37/37 [==============================] - ETA: 0s - loss: 0.3143 - accuracy: 0.8850 - auc: 0.9482 Epoch 33: val_loss did not improve from 0.30140 37/37 [==============================] - 4s 112ms/step - loss: 0.3143 - accuracy: 0.8850 - auc: 0.9482 - val_loss: 0.3020 - val_accuracy: 0.9050 - val_auc: 0.9588 Epoch 34/50 37/37 [==============================] - ETA: 0s - loss: 0.3101 - accuracy: 0.8874 - auc: 0.9490 Epoch 34: val_loss improved from 0.30140 to 0.28672, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 115ms/step - loss: 0.3101 - accuracy: 0.8874 - auc: 0.9490 - val_loss: 0.2867 - val_accuracy: 0.8949 - val_auc: 0.9610 Epoch 35/50 37/37 [==============================] - ETA: 0s - loss: 0.3106 - accuracy: 0.8840 - auc: 0.9491 Epoch 35: val_loss did not improve from 0.28672 37/37 [==============================] - 4s 117ms/step - loss: 0.3106 - accuracy: 0.8840 - auc: 0.9491 - val_loss: 0.2939 - val_accuracy: 0.8933 - val_auc: 0.9575 Epoch 36/50 37/37 [==============================] - ETA: 0s - loss: 0.2827 - accuracy: 0.9011 - auc: 0.9584 Epoch 36: val_loss improved from 0.28672 to 0.27233, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 109ms/step - loss: 0.2827 - accuracy: 0.9011 - auc: 0.9584 - val_loss: 0.2723 - val_accuracy: 0.9200 - val_auc: 0.9693 Epoch 37/50 37/37 [==============================] - ETA: 0s - loss: 0.3012 - accuracy: 0.8850 - auc: 0.9509 Epoch 37: val_loss did not improve from 0.27233 37/37 [==============================] - 4s 106ms/step - loss: 0.3012 - accuracy: 0.8850 - auc: 0.9509 - val_loss: 0.2929 - val_accuracy: 0.8849 - val_auc: 0.9552 Epoch 38/50 37/37 [==============================] - ETA: 0s - loss: 0.2847 - accuracy: 0.8946 - auc: 0.9554 Epoch 38: val_loss improved from 0.27233 to 0.25127, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 116ms/step - loss: 0.2847 - accuracy: 0.8946 - auc: 0.9554 - val_loss: 0.2513 - val_accuracy: 0.9147 - val_auc: 0.9689 Epoch 39/50 37/37 [==============================] - ETA: 0s - loss: 0.2759 - accuracy: 0.9007 - auc: 0.9597 Epoch 39: val_loss did not improve from 0.25127 37/37 [==============================] - 4s 112ms/step - loss: 0.2759 - accuracy: 0.9007 - auc: 0.9597 - val_loss: 0.2596 - val_accuracy: 0.9186 - val_auc: 0.9685 Epoch 40/50 37/37 [==============================] - ETA: 0s - loss: 0.2424 - accuracy: 0.9185 - auc: 0.9696 Epoch 40: val_loss did not improve from 0.25127 37/37 [==============================] - 4s 111ms/step - loss: 0.2424 - accuracy: 0.9185 - auc: 0.9696 - val_loss: 0.2652 - val_accuracy: 0.9008 - val_auc: 0.9639 Epoch 41/50 37/37 [==============================] - ETA: 0s - loss: 0.2696 - accuracy: 0.9019 - auc: 0.9607 Epoch 41: val_loss did not improve from 0.25127 37/37 [==============================] - 4s 113ms/step - loss: 0.2696 - accuracy: 0.9019 - auc: 0.9607 - val_loss: 0.2823 - val_accuracy: 0.8981 - val_auc: 0.9638 Epoch 42/50 37/37 [==============================] - ETA: 0s - loss: 0.2612 - accuracy: 0.9071 - auc: 0.9626 Epoch 42: val_loss improved from 0.25127 to 0.22276, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 116ms/step - loss: 0.2612 - accuracy: 0.9071 - auc: 0.9626 - val_loss: 0.2228 - val_accuracy: 0.9272 - val_auc: 0.9756 Epoch 43/50 37/37 [==============================] - ETA: 0s - loss: 0.2542 - accuracy: 0.9110 - auc: 0.9647 Epoch 43: val_loss did not improve from 0.22276 37/37 [==============================] - 4s 112ms/step - loss: 0.2542 - accuracy: 0.9110 - auc: 0.9647 - val_loss: 0.2529 - val_accuracy: 0.9143 - val_auc: 0.9696 Epoch 44/50 37/37 [==============================] - ETA: 0s - loss: 0.2647 - accuracy: 0.9041 - auc: 0.9588 Epoch 44: val_loss did not improve from 0.22276 37/37 [==============================] - 4s 108ms/step - loss: 0.2647 - accuracy: 0.9041 - auc: 0.9588 - val_loss: 0.2650 - val_accuracy: 0.8987 - val_auc: 0.9640 Epoch 45/50 37/37 [==============================] - ETA: 0s - loss: 0.2642 - accuracy: 0.9020 - auc: 0.9593 Epoch 45: val_loss improved from 0.22276 to 0.21198, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 118ms/step - loss: 0.2642 - accuracy: 0.9020 - auc: 0.9593 - val_loss: 0.2120 - val_accuracy: 0.9341 - val_auc: 0.9788 Epoch 46/50 37/37 [==============================] - ETA: 0s - loss: 0.2446 - accuracy: 0.9127 - auc: 0.9666 Epoch 46: val_loss did not improve from 0.21198 37/37 [==============================] - 4s 106ms/step - loss: 0.2446 - accuracy: 0.9127 - auc: 0.9666 - val_loss: 0.2237 - val_accuracy: 0.9204 - val_auc: 0.9743 Epoch 47/50 37/37 [==============================] - ETA: 0s - loss: 0.2389 - accuracy: 0.9178 - auc: 0.9675 Epoch 47: val_loss improved from 0.21198 to 0.20821, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 109ms/step - loss: 0.2389 - accuracy: 0.9178 - auc: 0.9675 - val_loss: 0.2082 - val_accuracy: 0.9263 - val_auc: 0.9774 Epoch 48/50 37/37 [==============================] - ETA: 0s - loss: 0.2286 - accuracy: 0.9188 - auc: 0.9707 Epoch 48: val_loss improved from 0.20821 to 0.19698, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_4.keras 37/37 [==============================] - 4s 119ms/step - loss: 0.2286 - accuracy: 0.9188 - auc: 0.9707 - val_loss: 0.1970 - val_accuracy: 0.9393 - val_auc: 0.9775 Epoch 49/50 37/37 [==============================] - ETA: 0s - loss: 0.2204 - accuracy: 0.9244 - auc: 0.9725 Epoch 49: val_loss did not improve from 0.19698 37/37 [==============================] - 4s 112ms/step - loss: 0.2204 - accuracy: 0.9244 - auc: 0.9725 - val_loss: 0.2285 - val_accuracy: 0.9239 - val_auc: 0.9730 Epoch 50/50 37/37 [==============================] - ETA: 0s - loss: 0.2372 - accuracy: 0.9144 - auc: 0.9681 Epoch 50: val_loss did not improve from 0.19698 37/37 [==============================] - 4s 112ms/step - loss: 0.2372 - accuracy: 0.9144 - auc: 0.9681 - val_loss: 0.2205 - val_accuracy: 0.9228 - val_auc: 0.9732 Training history saved to /content/drive/MyDrive/projects/school/Capstone Project/model_4_history.json
<keras.src.callbacks.History at 0x7bafb74dfaf0>
Plot the train and validation accuracy
plot_accuracy("model_4")
Plotting the classification report and confusion matrix
model_4_predictions = predictions("model_4", test_images_64x64, test_labels_64x64)
print("Classification Report:")
print(classification_report(test_labels_64x64, model_4_predictions))
Test accuracy: 0.9157692193984985
82/82 [==============================] - 0s 3ms/step
Classification Report:
precision recall f1-score support
0 0.91 0.91 0.91 1300
1 0.91 0.91 0.91 1300
accuracy 0.91 2600
macro avg 0.91 0.91 0.91 2600
weighted avg 0.91 0.91 0.91 2600
show_confusion(model_4_predictions, test_labels_8x8)
Now, let us try to use a pretrained model like VGG16 and check how it performs on our data.
model_vgg = VGG16(weights='imagenet', include_top=False, input_shape=(64, 64, 3))
model_vgg.summary() # displaying so I know what layer are there to pick from
Model: "vgg16"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_2 (InputLayer) [(None, 64, 64, 3)] 0
block1_conv1 (Conv2D) (None, 64, 64, 64) 1792
block1_conv2 (Conv2D) (None, 64, 64, 64) 36928
block1_pool (MaxPooling2D) (None, 32, 32, 64) 0
block2_conv1 (Conv2D) (None, 32, 32, 128) 73856
block2_conv2 (Conv2D) (None, 32, 32, 128) 147584
block2_pool (MaxPooling2D) (None, 16, 16, 128) 0
block3_conv1 (Conv2D) (None, 16, 16, 256) 295168
block3_conv2 (Conv2D) (None, 16, 16, 256) 590080
block3_conv3 (Conv2D) (None, 16, 16, 256) 590080
block3_pool (MaxPooling2D) (None, 8, 8, 256) 0
block4_conv1 (Conv2D) (None, 8, 8, 512) 1180160
block4_conv2 (Conv2D) (None, 8, 8, 512) 2359808
block4_conv3 (Conv2D) (None, 8, 8, 512) 2359808
block4_pool (MaxPooling2D) (None, 4, 4, 512) 0
block5_conv1 (Conv2D) (None, 4, 4, 512) 2359808
block5_conv2 (Conv2D) (None, 4, 4, 512) 2359808
block5_conv3 (Conv2D) (None, 4, 4, 512) 2359808
block5_pool (MaxPooling2D) (None, 2, 2, 512) 0
=================================================================
Total params: 14,714,688
Trainable params: 14,714,688
Non-trainable params: 0
_________________________________________________________________
I'm cutting last few layers from feature extraction part for two reasons:
transfer_layer = model_vgg.get_layer('block3_pool')
model_vgg = Model(inputs=model_vgg.input, outputs=transfer_layer.output)
for layer in model_vgg.layers:
layer.trainable = False
model_5 = Sequential()
model_5.add(model_vgg)
model_5.add(Flatten())
model_5.add(Dense(32, activation='relu'))
model_5.add(Dense(16, activation='relu'))
model_5.add(Dense(1, activation='sigmoid'))
# inside function compile_and_fit_model (DRY principle)
using callbacks
# inside function compile_and_fit_model (DRY principle)
Fit and Train the model
# It's extremely slow, so running just for 9 epochs to get the idea about how it goes
compile_and_fit_model(model_5, "model_5", train_images_64x64, train_labels_64x64, epochs=9)
Epoch 1/9 37/37 [==============================] - ETA: 0s - loss: 10.1576 - accuracy: 0.6500 - auc_3: 0.6644 Epoch 1: val_loss improved from inf to 1.49400, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_5.keras 37/37 [==============================] - 826s 22s/step - loss: 10.1576 - accuracy: 0.6500 - auc_3: 0.6644 - val_loss: 1.4940 - val_accuracy: 0.8189 - val_auc_3: 0.8661 Epoch 2/9 37/37 [==============================] - ETA: 0s - loss: 0.9483 - accuracy: 0.8441 - auc_3: 0.8949 Epoch 2: val_loss improved from 1.49400 to 0.53175, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_5.keras 37/37 [==============================] - 814s 22s/step - loss: 0.9483 - accuracy: 0.8441 - auc_3: 0.8949 - val_loss: 0.5318 - val_accuracy: 0.8651 - val_auc_3: 0.9289 Epoch 3/9 37/37 [==============================] - ETA: 0s - loss: 0.3933 - accuracy: 0.8854 - auc_3: 0.9456 Epoch 3: val_loss improved from 0.53175 to 0.39320, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_5.keras 37/37 [==============================] - 873s 24s/step - loss: 0.3933 - accuracy: 0.8854 - auc_3: 0.9456 - val_loss: 0.3932 - val_accuracy: 0.8777 - val_auc_3: 0.9493 Epoch 4/9 37/37 [==============================] - ETA: 0s - loss: 0.2809 - accuracy: 0.9033 - auc_3: 0.9610 Epoch 4: val_loss improved from 0.39320 to 0.32687, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_5.keras 37/37 [==============================] - 877s 24s/step - loss: 0.2809 - accuracy: 0.9033 - auc_3: 0.9610 - val_loss: 0.3269 - val_accuracy: 0.8910 - val_auc_3: 0.9553 Epoch 5/9 37/37 [==============================] - ETA: 0s - loss: 0.2414 - accuracy: 0.9132 - auc_3: 0.9678 Epoch 5: val_loss improved from 0.32687 to 0.29804, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_5.keras 37/37 [==============================] - 871s 24s/step - loss: 0.2414 - accuracy: 0.9132 - auc_3: 0.9678 - val_loss: 0.2980 - val_accuracy: 0.8925 - val_auc_3: 0.9614 Epoch 6/9 37/37 [==============================] - ETA: 0s - loss: 0.2110 - accuracy: 0.9221 - auc_3: 0.9734 Epoch 6: val_loss improved from 0.29804 to 0.27060, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_5.keras 37/37 [==============================] - 872s 24s/step - loss: 0.2110 - accuracy: 0.9221 - auc_3: 0.9734 - val_loss: 0.2706 - val_accuracy: 0.9024 - val_auc_3: 0.9623 Epoch 7/9 37/37 [==============================] - ETA: 0s - loss: 0.1648 - accuracy: 0.9378 - auc_3: 0.9820 Epoch 7: val_loss did not improve from 0.27060 37/37 [==============================] - 808s 22s/step - loss: 0.1648 - accuracy: 0.9378 - auc_3: 0.9820 - val_loss: 0.3433 - val_accuracy: 0.8913 - val_auc_3: 0.9608 Epoch 8/9 37/37 [==============================] - ETA: 0s - loss: 0.2480 - accuracy: 0.9070 - auc_3: 0.9669 Epoch 8: val_loss did not improve from 0.27060 37/37 [==============================] - 807s 22s/step - loss: 0.2480 - accuracy: 0.9070 - auc_3: 0.9669 - val_loss: 0.2987 - val_accuracy: 0.9048 - val_auc_3: 0.9630 Epoch 9/9 37/37 [==============================] - ETA: 0s - loss: 0.1690 - accuracy: 0.9356 - auc_3: 0.9818 Epoch 9: val_loss did not improve from 0.27060 37/37 [==============================] - 809s 22s/step - loss: 0.1690 - accuracy: 0.9356 - auc_3: 0.9818 - val_loss: 0.2956 - val_accuracy: 0.8997 - val_auc_3: 0.9631 Training history saved to /content/drive/MyDrive/projects/school/Capstone Project/model_5_history.json
<keras.callbacks.History at 0x7b452428cd00>
Plot the train and validation accuracy
plot_accuracy("model_5")
Model overfits a little.
model_5_predictions = predictions("model_5", test_images_64x64, test_labels_64x64)
Test accuracy: 0.885769248008728 82/82 [==============================] - 86s 1s/step
Plotting the classification report and confusion matrix
print("Classification Report:")
print(classification_report(test_labels_64x64, model_5_predictions))
Classification Report:
precision recall f1-score support
0 0.88 0.89 0.89 1300
1 0.89 0.88 0.88 1300
accuracy 0.89 2600
macro avg 0.89 0.89 0.89 2600
weighted avg 0.89 0.89 0.89 2600
show_confusion(model_5_predictions, test_labels_64x64)
Model 5 is surprisingly accurate considering it has weigths for features detection were trained on very, very different data, I was really expecting much worse performance. This shows how really good VGG16 architecture is, however it's still no go for our requirements, but that will be discussed in final section.
Let's see how our best model works if we scale data down, it may be important as market is very price sensitive and microscopes used there may not be top class.
model_6 = Sequential([
tf.keras.layers.Resizing(32, 32, 'bilinear'),
Conv2D(18, kernel_size=(3, 3), padding='same', input_shape=(32, 32, 3)),
LeakyReLU(alpha=0.25),
Conv2D(16, kernel_size=(3, 3), padding='same'),
LeakyReLU(alpha=0.20),
Conv2D(12, kernel_size=(3, 3), padding='same'),
LeakyReLU(alpha=0.15),
MaxPooling2D(pool_size=(2, 2)),
Conv2D(8, kernel_size=(3, 3), padding='same'),
LeakyReLU(alpha=0.10),
Flatten(),
Dense(32, activation='sigmoid'),
Dense(18, activation='sigmoid'),
Dense(1, activation='sigmoid')
])
compile_and_fit_model(model_6, "model_6",
images=train_images_64x64,
labels=train_labels_64x64,
validation_split=0.25,
epochs=25,
batch_size=512)
Epoch 1/25 37/37 [==============================] - ETA: 0s - loss: 0.6913 - accuracy: 0.5440 - auc_1: 0.5595 Epoch 1: val_loss improved from inf to 0.66571, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 7s 97ms/step - loss: 0.6913 - accuracy: 0.5440 - auc_1: 0.5595 - val_loss: 0.6657 - val_accuracy: 0.5984 - val_auc_1: 0.6728 Epoch 2/25 37/37 [==============================] - ETA: 0s - loss: 0.6012 - accuracy: 0.6971 - auc_1: 0.7444 Epoch 2: val_loss improved from 0.66571 to 0.52753, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 1s 41ms/step - loss: 0.6012 - accuracy: 0.6971 - auc_1: 0.7444 - val_loss: 0.5275 - val_accuracy: 0.7726 - val_auc_1: 0.7897 Epoch 3/25 35/37 [===========================>..] - ETA: 0s - loss: 0.4759 - accuracy: 0.8074 - auc_1: 0.8398 Epoch 3: val_loss improved from 0.52753 to 0.39865, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 2s 42ms/step - loss: 0.4720 - accuracy: 0.8101 - auc_1: 0.8449 - val_loss: 0.3987 - val_accuracy: 0.8585 - val_auc_1: 0.9123 Epoch 4/25 37/37 [==============================] - ETA: 0s - loss: 0.3555 - accuracy: 0.8821 - auc_1: 0.9153 Epoch 4: val_loss improved from 0.39865 to 0.32396, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 1s 40ms/step - loss: 0.3555 - accuracy: 0.8821 - auc_1: 0.9153 - val_loss: 0.3240 - val_accuracy: 0.9143 - val_auc_1: 0.9417 Epoch 5/25 35/37 [===========================>..] - ETA: 0s - loss: 0.2858 - accuracy: 0.9122 - auc_1: 0.9409 Epoch 5: val_loss improved from 0.32396 to 0.24121, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 2s 41ms/step - loss: 0.2850 - accuracy: 0.9123 - auc_1: 0.9410 - val_loss: 0.2412 - val_accuracy: 0.9288 - val_auc_1: 0.9559 Epoch 6/25 37/37 [==============================] - ETA: 0s - loss: 0.2165 - accuracy: 0.9405 - auc_1: 0.9639 Epoch 6: val_loss improved from 0.24121 to 0.17118, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 2s 44ms/step - loss: 0.2165 - accuracy: 0.9405 - auc_1: 0.9639 - val_loss: 0.1712 - val_accuracy: 0.9538 - val_auc_1: 0.9819 Epoch 7/25 37/37 [==============================] - ETA: 0s - loss: 0.1545 - accuracy: 0.9601 - auc_1: 0.9821 Epoch 7: val_loss improved from 0.17118 to 0.13184, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 2s 48ms/step - loss: 0.1545 - accuracy: 0.9601 - auc_1: 0.9821 - val_loss: 0.1318 - val_accuracy: 0.9667 - val_auc_1: 0.9877 Epoch 8/25 37/37 [==============================] - ETA: 0s - loss: 0.1290 - accuracy: 0.9662 - auc_1: 0.9868 Epoch 8: val_loss improved from 0.13184 to 0.12359, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 2s 44ms/step - loss: 0.1290 - accuracy: 0.9662 - auc_1: 0.9868 - val_loss: 0.1236 - val_accuracy: 0.9702 - val_auc_1: 0.9914 Epoch 9/25 35/37 [===========================>..] - ETA: 0s - loss: 0.1090 - accuracy: 0.9714 - auc_1: 0.9911 Epoch 9: val_loss improved from 0.12359 to 0.10032, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 2s 41ms/step - loss: 0.1087 - accuracy: 0.9714 - auc_1: 0.9912 - val_loss: 0.1003 - val_accuracy: 0.9720 - val_auc_1: 0.9931 Epoch 10/25 37/37 [==============================] - ETA: 0s - loss: 0.0988 - accuracy: 0.9740 - auc_1: 0.9926 Epoch 10: val_loss improved from 0.10032 to 0.09457, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 1s 38ms/step - loss: 0.0988 - accuracy: 0.9740 - auc_1: 0.9926 - val_loss: 0.0946 - val_accuracy: 0.9760 - val_auc_1: 0.9936 Epoch 11/25 37/37 [==============================] - ETA: 0s - loss: 0.0896 - accuracy: 0.9756 - auc_1: 0.9938 Epoch 11: val_loss improved from 0.09457 to 0.09003, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 2s 41ms/step - loss: 0.0896 - accuracy: 0.9756 - auc_1: 0.9938 - val_loss: 0.0900 - val_accuracy: 0.9744 - val_auc_1: 0.9943 Epoch 12/25 37/37 [==============================] - ETA: 0s - loss: 0.0921 - accuracy: 0.9746 - auc_1: 0.9926 Epoch 12: val_loss did not improve from 0.09003 37/37 [==============================] - 1s 40ms/step - loss: 0.0921 - accuracy: 0.9746 - auc_1: 0.9926 - val_loss: 0.0916 - val_accuracy: 0.9739 - val_auc_1: 0.9939 Epoch 13/25 37/37 [==============================] - ETA: 0s - loss: 0.0838 - accuracy: 0.9770 - auc_1: 0.9937 Epoch 13: val_loss did not improve from 0.09003 37/37 [==============================] - 1s 38ms/step - loss: 0.0838 - accuracy: 0.9770 - auc_1: 0.9937 - val_loss: 0.0940 - val_accuracy: 0.9742 - val_auc_1: 0.9922 Epoch 14/25 35/37 [===========================>..] - ETA: 0s - loss: 0.0776 - accuracy: 0.9779 - auc_1: 0.9949 Epoch 14: val_loss improved from 0.09003 to 0.08051, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 2s 41ms/step - loss: 0.0779 - accuracy: 0.9780 - auc_1: 0.9947 - val_loss: 0.0805 - val_accuracy: 0.9763 - val_auc_1: 0.9950 Epoch 15/25 37/37 [==============================] - ETA: 0s - loss: 0.0732 - accuracy: 0.9797 - auc_1: 0.9954 Epoch 15: val_loss improved from 0.08051 to 0.07746, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 2s 45ms/step - loss: 0.0732 - accuracy: 0.9797 - auc_1: 0.9954 - val_loss: 0.0775 - val_accuracy: 0.9771 - val_auc_1: 0.9963 Epoch 16/25 37/37 [==============================] - ETA: 0s - loss: 0.0705 - accuracy: 0.9800 - auc_1: 0.9960 Epoch 16: val_loss improved from 0.07746 to 0.07516, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 2s 49ms/step - loss: 0.0705 - accuracy: 0.9800 - auc_1: 0.9960 - val_loss: 0.0752 - val_accuracy: 0.9753 - val_auc_1: 0.9964 Epoch 17/25 35/37 [===========================>..] - ETA: 0s - loss: 0.0679 - accuracy: 0.9807 - auc_1: 0.9962 Epoch 17: val_loss improved from 0.07516 to 0.07339, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 1s 38ms/step - loss: 0.0681 - accuracy: 0.9808 - auc_1: 0.9960 - val_loss: 0.0734 - val_accuracy: 0.9772 - val_auc_1: 0.9960 Epoch 18/25 37/37 [==============================] - ETA: 0s - loss: 0.0637 - accuracy: 0.9810 - auc_1: 0.9965 Epoch 18: val_loss improved from 0.07339 to 0.07013, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 1s 38ms/step - loss: 0.0637 - accuracy: 0.9810 - auc_1: 0.9965 - val_loss: 0.0701 - val_accuracy: 0.9768 - val_auc_1: 0.9966 Epoch 19/25 37/37 [==============================] - ETA: 0s - loss: 0.0629 - accuracy: 0.9814 - auc_1: 0.9964 Epoch 19: val_loss improved from 0.07013 to 0.06969, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 1s 41ms/step - loss: 0.0629 - accuracy: 0.9814 - auc_1: 0.9964 - val_loss: 0.0697 - val_accuracy: 0.9766 - val_auc_1: 0.9967 Epoch 20/25 35/37 [===========================>..] - ETA: 0s - loss: 0.0624 - accuracy: 0.9820 - auc_1: 0.9963 Epoch 20: val_loss improved from 0.06969 to 0.06932, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 1s 39ms/step - loss: 0.0626 - accuracy: 0.9818 - auc_1: 0.9963 - val_loss: 0.0693 - val_accuracy: 0.9764 - val_auc_1: 0.9974 Epoch 21/25 35/37 [===========================>..] - ETA: 0s - loss: 0.0620 - accuracy: 0.9816 - auc_1: 0.9963 Epoch 21: val_loss did not improve from 0.06932 37/37 [==============================] - 1s 40ms/step - loss: 0.0627 - accuracy: 0.9813 - auc_1: 0.9962 - val_loss: 0.0816 - val_accuracy: 0.9752 - val_auc_1: 0.9942 Epoch 22/25 35/37 [===========================>..] - ETA: 0s - loss: 0.0617 - accuracy: 0.9811 - auc_1: 0.9967 Epoch 22: val_loss did not improve from 0.06932 37/37 [==============================] - 1s 40ms/step - loss: 0.0621 - accuracy: 0.9811 - auc_1: 0.9965 - val_loss: 0.0700 - val_accuracy: 0.9764 - val_auc_1: 0.9964 Epoch 23/25 37/37 [==============================] - ETA: 0s - loss: 0.0577 - accuracy: 0.9827 - auc_1: 0.9971 Epoch 23: val_loss improved from 0.06932 to 0.06605, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 2s 41ms/step - loss: 0.0577 - accuracy: 0.9827 - auc_1: 0.9971 - val_loss: 0.0660 - val_accuracy: 0.9774 - val_auc_1: 0.9962 Epoch 24/25 37/37 [==============================] - ETA: 0s - loss: 0.0565 - accuracy: 0.9823 - auc_1: 0.9969 Epoch 24: val_loss improved from 0.06605 to 0.06492, saving model to /content/drive/MyDrive/projects/school/Capstone Project/model_6.keras 37/37 [==============================] - 2s 49ms/step - loss: 0.0565 - accuracy: 0.9823 - auc_1: 0.9969 - val_loss: 0.0649 - val_accuracy: 0.9764 - val_auc_1: 0.9974 Epoch 25/25 36/37 [============================>.] - ETA: 0s - loss: 0.0564 - accuracy: 0.9825 - auc_1: 0.9968 Epoch 25: val_loss did not improve from 0.06492 37/37 [==============================] - 2s 44ms/step - loss: 0.0563 - accuracy: 0.9826 - auc_1: 0.9968 - val_loss: 0.0652 - val_accuracy: 0.9771 - val_auc_1: 0.9970 Training history saved to /content/drive/MyDrive/projects/school/Capstone Project/model_6_history.json
<keras.src.callbacks.History at 0x784bb03222f0>
model_6_predictions = predictions("model_6", test_images_64x64, test_labels_64x64)
Test accuracy: 0.9815384745597839 82/82 [==============================] - 0s 3ms/step
show_confusion(model_6_predictions, test_labels_64x64)
That is kinda surprising, but it's very fortunate that 32x32 is the way to go, that means we can use our tool for microscopes that have less magnification, which should be cheaper what is crucial for such price sensitive market.
mismatch_indices = np.where(model_6_predictions.flatten() != test_labels_64x64)[0]
misclassified_images = test_images_64x64[mismatch_indices]
misclassified_labels = test_labels_64x64[mismatch_indices]
fig, axes = plt.subplots(6, 8, figsize=(15, 10))
axes = axes.flatten()
for i in range(48):
axes[i].imshow(misclassified_images[i])
axes[i].axis('off')
label = healthy if misclassified_labels[i] < 0.5 else infected
axes[i].set_title(label, fontsize=8)
plt.tight_layout()
plt.show()
Looking in HSV it looks like many of those are just mislabeled, but that would need to be confirmed by professional diagnostician. Guessing that about half of them are mislabeled, whe can halve the error and claim we can reach over 99% accuracy, also taking into account that probably training data has mislabeled data. For such high accuracy, we may need to return mismatched data for relabeling and then run whole model again and check how it works. Still > 99% is secure claim.
All models are saved, and no model fits the goals, even if after relabeling mislabeled data we could improve accuracy to 99.9%. That would still be a no-go, because whole initial assumptions are wrong.
To detect malaria, we need to test well over a hundred cells (1% infected cells is a normal level of how malaria infects blood), so we need to detect a single infected cell in hundreds of images. Even 0.5% false positives will result in the classification of a really unacceptable amount of samples (sample being those hundreds of cells) as false positives.
Improvements that can be done:
We could try deep architecture like VGG16, but this time with retraining all weights and feature detection on data specific to our problem.
We could also improve model_6 which had 98% accuracy, but there is hard way to get better values without carefully fixing mislabeled data.
Yes, I'm using only HSV, as EDA has shown it's clearly the way to go.
The problem is solvable; initial experiments show computer vision is able to distinguish cells infected with malaria with very high accuracy (98%) even with data that contain some labeling errors.
Simple models are doing great job. Features are simple, it's not complex geometry that must be recognized.
Yes - there is still some scope, just don't use batch normalization and don't use data pretrained on detecting dogs, planes, and other unrelated objects.
None - we need to get back to business and, using gained insight, proceed with a different approach and different data.
We used the wrong data, that was clear from the start, but the results of this preliminary study are very promising and show that computer vision algorithms are able to detect infected cells with high accuracy and the project should be continued.
Therefore, we need to work with thick blood samples, and the provided data is not enough, as with thick blood samples we can't clearly separate single cells, and we need to work on whole images, not preprocessed images of single cells. Therefore, this whole experiment is a no-go, but it still provided some encouraging insights:
We shouldn't classify a single cell.
Let's say we aim for detecting 95% of infections, but for it to make sense, we need to limit false positives to 15%. This may cause concerns. Those are good results; some reference level is provided with much higher specificity but slightly lower sensitivity: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC84534/#:~:text=falciparum%20malaria%20by%20microscopy%20and,and%20a%20specificity%20of%2099.4%25: "While the test kit based on the detection of HRP-2 performed with a sensitivity of 92.5% and a specificity of 98.3%, the kit for the detection of pLDH showed a sensitivity of 88.5% and a specificity of 99.4%."
The typical ratio of infected cells is 1-5%; therefore, we need to detect a single cell in a sample where 0.01 cells are infected. Aiming for a 99.5% probability that such a cell will happen in the sample if someone is infected, the sample needs to contain over 500 cells. My conservative back-of-the-envelope calculation is somehow close to 200-500 cells that are checked as reported here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2481386/: "Although several methods for estimation of densities of blood-stage parasites by microscopy are in use, the most common is to count the number of asexual parasites seen relative to a given count of WBCs (usually 200 or 500 cells)"
import math
# 1 infected cell means presence of malaria
cells_in_sample = 350 # compromise here
sample_sensitivity = 0.95
sample_specificity = 0.85
def max_cell_false_negatives_rate():
# Calculating the maximum allowable probability of missing an infected cell
# so that at least one is detected with the desired sample sensitivity
# following Bernoulli Trials and the Binomial Distribution
return 1 - math.pow(1 - sample_sensitivity, 1/cells_in_sample)
def required_cell_sensitivity():
return 1 - max_cell_false_negatives_rate()
def max_cell_false_positives_rate():
# Calculating the maximum allowable probability of incorrectly identifying
# a non-infected cell as infected to maintain the desired sample specificity
# following Bernoulli Trials and the Binomial Distribution
return 1 - math.pow(sample_specificity, 1/cells_in_sample)
def required_cell_specificity():
return 1 - max_cell_false_positives_rate()
cell_sensitivity = required_cell_sensitivity()
cell_specificity = required_cell_specificity()
print(f"To achieve {sample_sensitivity} sensitivity and {sample_specificity} specificity on sample level")
print(f" we need specificity of {cell_specificity:5f} and sensitivity of {cell_sensitivity:5f} at cell level.")
To achieve 0.95 sensitivity and 0.85 specificity on sample level we need specificity of 0.999536 and sensitivity of 0.991477 at cell level.
That may be pursued but it will be hard to beat existing solutions but it is irrelevant for other reasons. Accuracy of 0.9955 is not so far away from our best of 0.981 accuracy for model_6 with mislabeled some of test data.
Anyway - for this project we need to work with whole samples, images/scans of whole thick blood smears - that would make it also more scalable, as the process of doing thin blood smears is complex and may provide less accurate data than in lab and it's not as easy, what is crucial considering target market with poor access to healthcare specialists that can prepare precise samples.
This application is local and lightweight, so virtually any computer can handle the task, keeping all sensitive data onsite to reduce the risk of data leaks. The only requirement is a reasonable internet connection, as a few megabytes or even a few dozen megabytes of data may need to be uploaded to the cloud for each sample.
This service may be offered by a cloud provider or hosted on-premises for organizations particularly focused on data privacy and who have the resources to deploy a reasonably powerful server.